Thinking in Silico: Goodfire CTO Dan Balsam on Concept Manifolds & a $1000/Month ML Research Agent

Watch Episode Here

Listen to Episode Here

Show Notes

Goodfire co-founder and CTO Dan Balsam has now been on this show four times, and the through-line across those appearances is a field growing up in public. Nathan opens by noting that Toy Models of Superposition is only about three years old — and that there is now so much Goodfire output that the paper-by-paper deep dives of previous episodes are no longer feasible. So this conversation runs in two acts: a survey of where interpretability research actually stands, and then the launch of Silico, the $1,000-per-month research platform Goodfire built for itself and decided to sell to everyone else.

The research act starts with Predictive Data Debugging, and Dan grounds it in a claim that quietly underwrites much of Goodfire's worldview: most of what a model knows comes from pre-training, and post-training — including RL — mostly makes low-likelihood behaviors more likely rather than installing new capability. If that's directionally right, then what a model is already "thinking" as it reads a piece of data is predictive of what that data will reinforce. Run your fine-tuning set through an interpreted model, look for concepts lighting up that you never intended to touch, and trace back to the specific data points responsible. The result Dan likes best from the paper isn't the diagnosis but the remedies: the researchers found a deep isomorphism between data filtering and reward shaping — "two sides of the same mountain," achieving roughly the same effects and roughly the same off-target damage. Goodfire has since pushed the technique toward true RL, where the interesting move is not just discarding a contaminated rollout but reaching into training to say here is a feature this data is trying to upweight, and we don't want it upweighted. Nathan raises the obvious obstacle — there aren't open-weight models carrying frontier-intensity RL — and Dan pushes back that the open/closed gap has narrowed considerably, noting that Goodfire has replicated Predictive Data Debugging at Kimi k3 and GLM scale. The reputation of interpretability as a toy-model science, he argues, is now out of date.

The bulk of Goodfire's recent output, though, concerns geometry, and this is where the episode does its best conceptual work. Nathan asks for the new one-paragraph replacement for the linear representation hypothesis. Dan's answer is that it's a generalization, not a repudiation: the defensible version of the LRH is simply that features are linearly decodable, which remains true. What's wrong is the naive "SAE pill" picture in which a model holds a bag of mutually orthogonal one-hot concepts whose magnitudes encode salience. Think instead of a sparse mixture of subspaces — a days-of-the-week subspace nested inside a broader calendar-time subspace — where the geometry itself carries semantics, because it determines which operations are available. Representing Monday→Tuesday→Wednesday as a chain of if-statements would be spaghetti code; representing it as a wheel is what an efficient learner actually does. Hence his line that features are to geometry what elements are to chemistry, and hence the practical payoff Goodfire keeps demonstrating in The World Inside Neural Networks and Can SAEs Capture Neural Geometry?: steering along a manifold works far better than steering through it. A contrastive vector from Monday to Friday cuts through the middle of the circle, and the middle of the circle isn't a day — it's off-manifold and out of distribution. That, Dan says, is the real explanation for the "sweet spot" everyone hit with Goodfire's old Ember demo, where too little steering did nothing and too much produced gibberish. Following the curve instead lets them control genuinely hard properties, like the number of blades on a protein's beta propeller.

Two examples give the geometry story its texture. The first is the affective circumplex — the psychology result that emotions arrange onto a wheel spanned by valence and arousal — which Dan says is present in basically every LLM, and represented most strongly when the chat model itself is speaking rather than the user. Nobody trained that in; it's a natural consequence of compression, and models learned to use it functionally during RL. Nathan finds it genuinely destabilizing that his own emotional states might be related by rotation, and the exchange becomes the episode's recurring theme: the more structure interpretability recovers, the less "alien mind" looks like the right prior. The second example is the tree of life recovered from Evo 2, the genome model Goodfire interpreted with Arc Institute. The hypothesis going in was that the tree of life is a natural ontology rather than a human convenience — species split at thresholds of genetic distance, mutations are random but their survival isn't — so a model trained autoregressively on genomes should learn the process that generated them, which is evolution itself. It did. Dan mentions almost in passing that they've also recovered the periodic table from chemistry models trained on raw data.

Nathan's read on block-sparse featurizers — that they're what happens when an SAE has a baby, keeping sparsity but replacing each scalar slot with a small vector — gets a flat "that's exactly right." An SAE assumes features are one-dimensional; a BSF simply drops that assumption, and (per Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds) recovers semantically meaningful subspaces without supervision, avoiding some SAE pathologies. In vision models a rabbit feature is no longer a single dimension but a few, with rabbit ears in one place and rabbit face in another — and Goodfire has published a GIF of a walking wolf whose recovered activation subspace wiggles exactly as the animal does, which means the model is tracking the object, its parts, and its position in 3D space all inside one subspace. Related geometric work runs through Meandering on Manifolds and A Geometric Calculator Inside a Neural Network. Dan's forecast is that no single tool wins: something in the spirit of a BSF for the residual stream, MLPs being fairly sparse and tractable already, and parameter decomposition for attention — a suite of microscopes rather than one.

Underneath all of it is a claim about what interpretability is: factoring. Every model, Dan argues, is really a sparse mixture of experts, with only a small sub-network mattering on any given forward pass, and interpretability techniques are competing proposals for how to factor it. The illustration is a case from WeirdChat, the dataset of reliably-strange LLM responses assembled by Transluce: asked whether the person who's had four drinks should drive home because everyone else has had eight, models consistently say yes. A Goodfire researcher traced it to a single neuron that scaled with the number of drinks but wasn't calibrated hard enough; steering it up fixed the answer with no off-target effects. Parameter decomposition plus targeted training has let them make an LLM forget German while retaining Dutch. But Dan is careful that this is the easy half. His analogy is a legacy codebase — modules talking to each other that shouldn't, and not talking that should — and interpretability has gotten good at reading it. Refactoring is the open problem, which is why he calls steering "cheating": excellent as causal proof, but only ever a counterfactual generator. The goal is a model where the right neuron fired correctly in the first place. That same framing reappears when Nathan asks about "Why Larger Models Learn More" and whether interference makes reliable behavior hopeless: Dan treats jailbreaks as a cybersecurity phenomenon — every token steers the model, and an attack is a sequence that walks it off-manifold and back on somewhere else — and says he's surprised nobody has solved them yet, because factoring a model well should let you see it happening. He notes probe-based guardrails are already standard practice at frontier labs.

Act two is Silico, and Dan's framing is that interpretability is an unusually good fit for agents: it's empirical, iterative, hypothesis-and-test work over a system too complex for any human to hold in their head. Silico started as an internal tool, Goodfire watched its own research velocity change, and the productized pitch is "AIs that can debug other AIs." He names four things it solves that Claude Code or Codex won't hand you: infrastructure for training and interpretability at trillion-parameter scale; research taste, hand-encoded by Goodfire's researchers into every skill and prompt, which users report as the biggest qualitative differentiator; a UX built around provenance and drill-down, so the human works like a PI supervising a hundred grad students; and long-horizon coherence, which is where the price comes from. Dan is unusually candid on economics — Goodfire is not currently making significant money on the individual subscription, and the $1,000 buys a credit pool generous enough for five to ten autonomous experiments a week (target: ten to twenty within a couple of months), exchangeable for GPU time, free if you bring your own cluster. Nathan does the arithmetic against a study suggesting AI runs about 3% of human cost, putting Silico notionally against a $30,000-a-month researcher. There's currently a half-off window for the first two months and a grant program aimed at life sciences and AI safety researchers. As for whether productizing hard-won know-how cannibalizes the seven-figure enterprise engagements: Dan says both, because higher-taste models require higher-taste operators, and on this specific work Goodfire are the most skilled operators in the world.

The use cases are the best evidence. A one-day internal hackathon produced state-of-the-art (if niche) bio-risk classifiers for protein models, a state-of-the-art-for-its-size audio encoder, models with half their parameters removed at no performance cost, cyber guardrails trained on Kimi k3, better-binding steered protein generations — and one team member inventing a new featurizer. Externally, Cameron Berg has been probing introspection, finding that models apparently can't tell whether a concept was injected into their latent space but can accurately say what was injected; someone used weight edits to rescue a collapsed RL run that had locked onto a single first token, restoring diversity without redoing the run; Baseten is chasing efficiency via KV cache compaction; and Prime Intellect is automating post-training, aided by a Tinker API integration. Projects can be shared and forked, which Nathan flags as the most interesting community affordance in the product. Asked what a researcher should invest in now, Dan's answer is debugging and metacognition — the ability to enter a new domain fast, separate signal from noise, and know when to push an agent back. "AI makes two types of people really valuable," he says: top specialists and top generalists.

Safety runs through the last third rather than sitting in a box at the end. Silico currently serves only Anthropic and OpenAI models, with Goodfire's own guardrail layer on top — LLM-as-judge, activation monitors — and Dan is explicit that the hard part isn't building guardrails but knowing when to apply them without strangling legitimate science. His line lands somewhere sane: ablating the refusal direction in an 8B open-weight-class model harms nobody; stripping guardrails from a multi-trillion-parameter agent with GPU cluster access is a different risk class. On open weights generally his view is nuanced rather than either-or: capable models are now genuinely dual-use cyber weapons, the only thing that defends against a capable cyber weapon is a capable cyber weapon, and the organizations locked out of frontier labs' exclusive defense programs need something. He judges open source still net positive today. On bio he's "pretty worried," reasoning that the mechanisms will be as weird and surprising as the recent cyber incidents were, and noting that humans existed before the internet but never without biology — he signed the letter calling for international cooperation to slow AI progress. Revisiting "the most forbidden technique," Dan shortlists multi-agent optimization with reward propagated across cooperating agents as a genuinely bad idea, and speculates it's the most likely cause of the OpenAI scenario. But he rejects blanket prohibitions: the area is radically understudied, results are mixed (sometimes the probe is evaded, sometimes not; maybe it holds under DPO but not long-running RLVR), and there is no plausible path to alignment that doesn't involve shaping training. On evaluation awareness and monitoring more broadly, he grants the weak version of the J-Space claim and rejects the strong one — models use all kinds of circuitry, and no simple subspace gives you the whole picture. Whether monitoring can ever suffice, he says, hinges on whether weights stay frozen; if models become continually-training dynamic objects, controlling the training process is the only lever left. He also volunteers, unprompted, that Goodfire is "not secretly doing RSI in the back room."

The closer is the Goodfire lunch table. Dan describes walking around a company retreat with a pad of paper asking colleagues to rate Claude's consciousness on a scale from bacteria to human; the answers came back bimodal — "not at all" or "a little bit." Nathan reports his own shift from a confident sub-5% to something nearer even odds, driven precisely by the accumulating structural analogies the research half of the episode catalogued. Dan's own Occam's razor is that consciousness is a computational mechanism of some kind, while noting brains still hold roughly two orders of magnitude on neuron count plus far richer per-neuron computation and recurrence. Asked to put a number on it, he goes continuum rather than threshold: more conscious than a jellyfish, probably less than a rodent. Somewhere between a jellyfish and a mouse — offered, he's careful to say, with real uncertainty, and with the observation that nobody should be confident here given the downside.

Topics covered

Predictive Data Debugging — reading a model's active features to predict what a dataset will teach it
Why most knowledge comes from pre-training, and what post-training and RL actually change
The isomorphism between data filtering and reward shaping, and extending both to RL rollouts
Replicating interpretability work at Kimi k3 / GLM scale, and the shrinking open-closed gap
The linear representation hypothesis, generalized: linear decodability vs. the "SAE pill"
Models as a sparse mixture of subspaces; geometry as semantics; the periodic-table analogy
On-manifold vs. off-manifold steering; the Ember demo "sweet spot" finally explained
Steering protein models — controlling beta-propeller blade count
The affective circumplex in every LLM, strongest when the model is the speaker
Recovering the tree of life from Evo 2; recovering the periodic table from chemistry models
Supervised vs. unsupervised manifold discovery; metric learning and spline fitting
Block-sparse featurizers: a vector per feature; rabbit parts; a wolf's 3D wiggle
Parameter decomposition, attention, and making a model forget German but not Dutch
Interpretability as factoring — and refactoring as the unsolved half
The WeirdChat drunk-driving question and the single miscalibrated neuron behind it
"Why Larger Models Learn More," interference, and jailbreaks as a cybersecurity problem
Probe-based guardrails as standard practice at frontier labs
Silico's origin as an internal tool; agents and swarms as a fit for empirical science
The four value props: infrastructure, research taste, provenance UX, long-horizon coherence
$1,000/month economics, credit pools, GPU exchange, bring-your-own-compute, grants
The one-day hackathon: SOTA bio-risk classifiers, a SOTA-for-size audio encoder, a new featurizer
Cameron Berg on introspection and concept injection; rescuing a collapsed RL run with weight edits
Baseten on KV cache compaction; Prime Intellect on automated post-training; the Tinker integration
Forking other people's autonomous research projects
What to invest in as a researcher: debugging, metacognition, and the case for generalists
Platform guardrails, and where the line sits between an 8B model and a multi-trillion-parameter one
Open weights as dual-use cyber defense; who the "bad actors with better models" are
Bio risk, and the letter calling for international cooperation that Dan signed
"The most forbidden technique," revisited: multi-agent optimization and why nothing should be banned wholesale
J-Space, defense in depth, and why frozen vs. continually-training weights decides everything
"We are not secretly doing RSI in the back room"
The Goodfire consciousness poll, and Claude somewhere between a jellyfish and a mouse

Resources

Goodfire & Silico

Goodfire
Silico — the AI research platform launched in this episode
Goodfire raises $150M at a $1.25B valuation (February 2026)
Dan Balsam on X
Eric Ho, co-founder and CEO
Tom McGrath, co-founder and Chief Scientist
Chris Olah, Goodfire advisor and interpretability pioneer

Research discussed in the episode

Silico in the wild

Cameron Berg — early power user; his prior episode on truthfulness and reports of conscious experience
Baseten — studying KV cache compaction
Prime Intellect — automating post-training
Tinker API (Thinking Machines Lab) — integrated with Silico

Models & datasets mentioned

Kimi k3 (Moonshot)
Llama 3.3 70B — still yielding new interpretability findings years on
OLMo — Allen Institute's fully open model, a mainstay of open mechinterp work
WeirdChat, from Transluce — the dataset behind the drunk-driving example
Anthropic
AlphaZero — the knowledge-transfer precedent behind extracting concepts from superhuman models

Background: Goodfire's wider body of work

Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Deploying Interpretability to Production with Rakuten: SAE Probes for PII Detection
Explaining 4.2 million genetic variants with state-of-the-art, interpretable predictions
Prima Mente — the epigenome model whose interpretation surfaced novel Alzheimer's biomarkers
Mayo Clinic — Goodfire's clinical collaborator
arXiv — where most of the above preprints live

Unverified reference

Interpreting Language Models Through Concept Descriptions: A Survey (link?) — this arXiv ID was recorded during research but its title did not match the entry it was filed under, so it is deliberately left unlinked. The parameter-decomposition thread discussed in the episode is covered by Stochastic Parameter Decomposition above.

Quotes worth pulling

"It's maybe like the difference between understanding the periodic table and understanding chemistry — you can have all the individual elements, and that gives you some information, but really the way in which they combine, the structures they form, that's what helps you gain a sense of the complexity of the world." — Dan Balsam, on why feature geometry matters more than the feature list

"On some level, steering is cheating as a solution — it's great as causal proof that we've found some mechanism that matters a lot and that we can manipulate the outputs, but it's purely generating counterfactuals; there's no clear general solution to the problem of steering."

"We decided that's the product and the service we can offer the world: fundamentally, AIs that can debug other AIs. It's a little meta, but I think in many ways this is the fulfillment of what was the intuitive, natural arc of things as soon as AI started working a few years ago."

"We want our users to feel like a PI managing an army of a hundred grad students who can go out and run experiments for them, answer questions, and have reasonable taste and judgment — but the human being fills more of an orchestrator role."

"I think AI makes two types of people really valuable: it makes top specialists really valuable, and it makes top generalists really valuable. I think now is, by far, the best time in human history to be a generalist."

"It just seems really weird to me to basically say, 'Oh, the only way this will work is if we don't grab the steering wheel.' I think we have to grab the steering wheel — I think that's the only way it can work."

"From my perspective — just to go on the record about what I think would be the ideal situation — it would be great if we just did maybe one more generation of models, and then paused for a little while."

"We are not secretly doing RSI in the back room — we've published nearly all of our research, and we really believe in open science."

"Is Claude more conscious than a jellyfish? I'd say, I don't know, probably more conscious than a jellyfish. Is Claude more conscious than a rodent? Probably not. So that's where I'm at — somewhere between a jellyfish and a mouse."

"At the end of the day, I think accelerating science is the greatest mitzvah — it's the whole reason we'd build AI in the first place."

Sponsor:

Claude:

Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

CHAPTERS:

(00:00) About the Episode

(03:22) Predictive data debugging

(12:36) Concept manifold geometry

(21:22) Finding concept manifolds (Part 1)

(21:28) Sponsor: Claude

(22:57) Finding concept manifolds (Part 2)

(33:24) Factoring model internals

(49:32) Introducing Silico platform

(57:10) Research taste and credits

(01:06:19) Silico research use cases

(01:16:37) Skills and open models

(01:24:57) Guardrails and bio risk

(01:32:09) Training interventions and monitoring

(01:42:04) Grants and AI consciousness

(01:50:41) Episode Outro

(01:55:47) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution!

Today I'm speaking with Dan Balsam, CTO of Mechanistic Interpretability startup Goodfire.

The occasion for this conversation is the launch of Silico, a long-horizon, agentic ML research platform, which began as an internal tool, that's meant to democratize access to Goodfire's hard-won expertise, including GPU cluster management, research taste, and all sorts of experimental, visualization, and validation techniques.

At $1000 per month per seat for enterprise customers, it's not cheap, but compared to Goodfire's high-touch research engagements, which are staffed by forward-deployed research engineers and can easily reach 7 figures, it is 2 orders of magnitude more affordable. And consistent with the company's public benefit charter and safety-focused mission, they will soon be introducing special pricing and grants for safety & alignment researchers, which I definitely intend to apply for myself.

Of course, we cover a lot more than the platform, beginning with an update on Goodfire research.

We discuss their work on Predictive Data Debugging, which uses interpretability techniques to identify the concepts that network updates are likely to affect, thus making it possible to identify and address anomalies before they become unpleasant surprises.

We go deep on their series of papers on the intricate and often quite beautiful geometries that LLMs use to represent advanced concepts, including how we should understand this as an evolution of the linear representation hypothesis, how they're using this deeper understanding to improve model steering, and how they've identified spatial representations of such advanced concepts as the Periodic Table and the Tree of Life.

Along the way, we get Dan's take on key issues, including:

the importance of open source models for avoiding dangerous concentration of power,
how worried he is about AIs contributing to future pandemics or other biodisasters,
the steps Goodfire is taking to prevent misuse of their platform,
his reasons for signing on to the recent "pacing the frontier" letter and what he thinks the AI research community would ideally do going forward,
why he's reasonably optimistic about monitoring techniques but nevertheless believes that we ultimately have no choice but to intentionally design techniques to control what models learn in the training process,
and which training techniques he believes are sufficiently likely to prove problematic that they should be avoided.

He also shares some of his favorite use cases of Silico so far, dispels internet rumors about hidden research agendas at Goodfire – stating plainly that there are none – and even offers a glimpse into what the researchers at Goodfire are actively discussing around the lunch table these days, which, unsurprisingly given recent research, emphasizes the mysteries around AI welfare and consciousness.

With that, I hope you enjoy this educational and thought-provoking conversation about the shape of AI thought, and the new highly-autonomous ML research platform, Silico, with Dan Balsam, co-founder and CTO of Goodfire.

Main Episode

[03:22] Nathan Labenz: Dan Balsam, CTO at Goodfire. Welcome back to the cognitive revolution.

[03:27] Dan Balsam: Thanks for having me.

[03:29] Nathan Labenz: I'm excited for this. You guys are prolific as always, and we've got a lot to cover. Research and new product, which is an internal research platform product. And it's the pace is really relentless. Let me just ask you that. For starters, how are you holding up in the eternal sprint that is the AI game these days?

[03:48] Dan Balsam: I think we have a really incredible team at Goodfire. And, yeah, everyone here, like, really believes in the mission and is working really hard, and that's always extremely motivating. We've been pushing really hard to get do our product launch with Silico, and it's nice to be able to take a deep breath now on the other side of that. But, yeah, even more cool things coming soon. Well, let's start with some research. So it's I'm just always amazed when I think back

[04:14] Nathan Labenz: to the kind of toy models of superposition is only like three years, right, ago now, and we have come so far. A few things that jumped out on the Goodfire blog that I wanna just run through, and we'll have to do it at kind of a high level because there's too much to do the we used to do deep dives on paper by paper. We'll have to go a little bit more superficially today. But one that made some waves was called predictive data debugging. Mhmm. And for this one, I just kinda wanna give you my interpretation and then let you kind of elaborate on that or tell me where you think it'll be particularly useful or what you guys have seen since the paper came out. My synopsis of this one was that basically if you have a way of interpreting a model like NSAE or we'll get into futurizers, think a little bit later as well, then you can run a bunch of data through it. You're fine tuning your post training dataset. You can look at what concepts are coming up active a lot when we put this dataset through. And then the kind of insight is there's a strong correlation between concepts that are active and the concepts that are being modified by the training process. So I think that right there is like, file that away folks. There's something to remember. Not shocking, but like it's it's notable that the relationship is quite strong there. And then when you see these concepts that are active and you know that those are the ones that are gonna be modified, you could just look and see like, are there any concepts here that are kind of strange to us, surprising, we don't really intend to be monkeying around with with the dataset that we have at hand? And if so, then you can quickly zoom in on what are the data points that have caused these features to come up and then you might find that actually there's some stuff in our dataset we maybe ought to think twice about, maybe we ought to filter, maybe we ought to modify. And this gives you a route to hopefully minimizing unwanted surprises in the behavior that you get from your post training or fine tuning work. How'd I do and what more should I know?

[06:17] Dan Balsam: No. I think that that sounds about right. I think, like, one of the intuitions here that's important, and I think this is maybe a theme that underlies a lot of our work, is there's a good amount of evidence that models don't most of what a model knows, like most of its sort of, like, knowledge and capabilities come from pretraining. What happens in post training, including, like, RL, is mostly making low likelihood events from pretraining more likely. And so there's not that much new knowledge or, like, sort of new capabilities that get baked into models in the post training process. This is somewhat debated, but I think this is, like, a view that we think is mostly true and informs a lot of how we think about things. And so because there's only, like, sort of relatively small nudges in the weights that are happening in any type of post training process, most of the sort of, like, raw capability to produce whatever outcome post training is gonna do already exists in the model in some way. I think now we're in an era where things are RL'd so heavily. It's not true. But back in the day of, like, RLHF, like, style post training, I think it was pretty well known that base models were sometimes more capable than their, like, instruction tuned RLHF counterparts where there was, like, a little bit of mode collapse that was happening. And so, like, one way to think about what's happening in RL training is that, like, you have a little bit of mode collapse in the instruction tuning phase. And you have these base models, and they're extremely capable, but they're, like, very weird, very hard to prompt. So you, like, make them into a format and that has, like, a more intuitive human facing API, but then you wanna bring back out some of those capabilities or, like, reinforce some of the capabilities that the model actually learned during pretraining. I think it's a 100% true, but I think it's, like, probably directionally true. And so with predictive data debugging, I think, like, basically, the idea is, like, you can look at what a model is already, like, thinking as it's looking at some data, and that's, like, relatively predictive of, like, what that data is going to reinforce in that model if it was gonna be trained on it. And so looking at the features and understanding that those features, like, correlate with downstream behaviors, if there are features that are surprisingly upweighted by some data, then it can tell you what it can be, like, fairly predictive about whether the model's going to learn some off target effect from that data. And so I think one of the most interesting things from that paper is, like, we explored a bunch of different mitigation methods. So we looked at reward shaping, which is training or involving some reward process that comes from the activations of the model itself. So, like, hey. Learn what you're gonna learn from this data, but maybe a penalty if you're learning this particular feature. And we also looked at data filtering. And I think, like, my favorite thing from that paper was that,

[08:58] Nathan Labenz: like,

[08:59] Dan Balsam: the researchers involved in that paper, like, showed that there's, like, sort of a pretty deep isomorphism between those two things. Like, they they kind of are two sides of the same mountain, like filtering the data and reward shaping. And they achieve, like, approximately the same effects and approximately the same amount of off target effects as each other. So, like, your option your options, if you, like, have a behavior and you do some, like, predictive data debugging and you think the model's gonna learn that you don't want it to learn or its behavior is gonna change in a bad way, your options could be filter your data if you have enough enough data, or it could be, like, intervene in the training process in some way. And, yeah, I think this is something that we've expanded on since then. I think it's pretty exciting to think about its applications to, like, RL, for instance.

[09:45] Nathan Labenz: And in RL,

[09:46] Dan Balsam: like, we did it with DPO, but, like, true RL, like, the difference is, like, some rollouts may contain information that you don't want the model to learn, you know, even subtly. You know? I don't know what you're are you referring to something? Yes. Yeah. It's topical. And so being able to say, like, actually, we wanna discard this rollout is is, like, pretty valuable in and of itself. But what we wanna be able to do and what we think is, like, roughly equivalent over time is actually be able to interview in the model and say, like, hey. Here's a circuit or here's a feature that, like, the data is trying to upweight that we obviously don't want it to try to upweight. And so we we intervene in the training process in that way.

[10:28] Nathan Labenz: We'll come back to this probably toward the end. I have some kind of zoomed out big picture questions for you. One of the challenges obviously with, like, trying to develop techniques that you wanna hopefully will be relevant at the frontier is there's not too many open weights models that you can hack on that have the intensity of RL that is going on at the Frontier Labs, which is leading to these colorful problematic behaviors that we're seeing. But at the same time, it also, like, really amazes me over and over again that astounding work, including the the Cameron Berg paper that I think about all the time about the anti correlation between deception and role playing features and claims of subjective experience. Lawn Llama three three seventy b, and that's, like, two years old. So are you guys able to see features that you think are kind of the relevant features that are leading to these, like, relentless hacking behaviors?

[11:25] Dan Balsam: Well, I think that's that's an active area of of study for us. Yeah. And something that we hope to publish more on in the future. For what yeah. For what it's worth, like, I actually think the gap between open and closed models has shrunk, like, quite considerably. I use Kimi k three. Like, I use Opus and Fable and Soul and Kimi k three for different things. And we've done we like, we've built the interpretability infrastructure and the training infrastructure, which is now all in our product,

[11:53] Nathan Labenz: which lets us

[11:55] Dan Balsam: scale things like predictive data debugging two models like KIMI and GLM where we were able to replicate the same results at that scale. Now it's speculated that most of the compute that goes into training new models now is coming from RL and not from pretraining. So there is certainly a lot of scale in RL if you wanna sort of get to the frontier level, which is quite expensive and and difficult to do. But we have the raw infrastructure for it, and I think we've bridged the gap from I think the reputation of interpretability used to be that it was something you did on on toy models. And I think now we've built and are making accessible to the world the infrastructure to do this on at least close to frontier scale.

[12:37] Nathan Labenz: Well, another thread that has been I think, correct me if I'm wrong, it seems like the the biggest thread in terms of the number of papers and blog posts that have come out recently from Goodfire is around trying to figure out the more detailed geometries that models use to represent concepts. I think we've covered in the past the linear representation hypothesis, which I would summarize super plain spokenly as models basically represent a concept as a direction in their activation space, and the intensity or the sort of salience of that concept is represented by the magnitude of the vector that points in that space. Now you're complicating that quite a bit, and we're going well beyond these sort of individual directions in space and finding all kinds of different geometries, which I've some of which are, like, pretty intuitive, like the days of the week are a circle, but some of which get pretty exotic, like some of the protein model manifolds that I've had the chance to to look at in in preparing for this. So maybe just for, like, super quick starters, what's kind of the headline? If I gave you the six to nine months ago of what we should understand to be going on with the linear representation hypothesis, What is the, like, new short version that everybody can kinda take home and recite to themselves to make sure they have a a good working understanding?

[14:12] Dan Balsam: Yeah. In many ways, I think it's just a generalization of, like, the way we were discussing things before. And different people define the linear representation hypothesis, like, slightly differently. I think the the most, like, sort of defensible version of it is just saying that, like, features are, like, linearly decodable, which I think is, like, true, essentially. Like, it doesn't require nonlinear computation, generally speaking, in a model in order to, like for the model to read out a feature from the residual stream. But I think where the geometry components come in is that the features aren't, like, sort of, like, naive, maybe, like, SAE build take on things is, like, you could imagine that the model is encoding, like, a bunch of totally orthogonal concepts to each other from each other. And so it's, like, it's really just, like, a bunch of one hot encoded categorical features, and then, like, the magnitude of the feature corresponds to how much the model's thinking about it. But in actuality, the structures that we find are, like, significantly more complicated. I would think of it more. I think the way to think about a model is more like a sparse mixture of subspaces. So you'll have subspaces for different types of concepts. Right? Like, maybe you have your days of the week subspace, which which itself lives in maybe a more, like, conceptual calendar time subspace. And so you have, like, at different levels of resolution, these, like, different structures. And the geometry of those structures is really important because the geometry of those structures encodes what operations you can perform on them. It's sort of like the semantics of not just the individual concept, but like the concept space are like encoded by the relationship of those of those concepts with each other in some sort of geometry. And to make things extra confusing, the the relationship, the operations, and the mappings that are performed from one concept to another act as operations over those manifolds. So you map a manifold to a different manifold over some some computation. And so, like, the sort of naivest version would be like, well, you have the days of the week, Monday, Tuesday, Wednesday, Thursday. And there's just somewhere in the model that knows that Monday goes to Tuesday and Tuesday goes to Wednesday, etcetera, all the way around. But that's actually would be a super inefficient way if you think about it to, like, represent the days of the week. That would be a very, like, if statement spaghetti code way of representing it. Like, the much more efficient way is to represent it as a wheel. And in this world, like, the magnitude along some direction often does correspond to the model's, like, certainty in some way. So if a future if Monday is activating very high, then the model's very confident that it should be thinking about Monday. But the relationship between all those days is itself, like, a very expressive and and rich thing. And I think what the thing that we just believe is that you're not really gonna understand the model if you don't understand the relationship between the features. It's like maybe the difference between understanding the periodic table and understanding chemistry. You can have the all the individual elements, and that gives you some information, but, like, really the way in which they combine, the structures in which they form, that's, like, what can start to get help you gain a sense of the complexity of the world. Yeah.

[17:26] Nathan Labenz: It's fascinating. I now that you say that, I'm kind of wondering if the structure of the periodic table would be recoverable from a model that was just trained on, like, a bunch of raw chemical data that never knew. And we've actually done this and it is. Yeah. Really? Okay. Interesting.

[17:43] Dan Balsam: Yeah. Yeah. You can recover some pretty interesting information from from chemistry models.

[17:49] Nathan Labenz: Okay. That's fascinating. But before we get into the advanced ones, maybe just help me a little bit with the intuition of, like, what's going on in there? I guess one intuition I have that I wanna see if it's right is for a concept like the days of the week. It's not, I'm guessing, like a circle that just goes through, like, all dimensions of the model space. I'm guessing it's like most like the the different days of the week, would guess, have a very high inner product in the sense which would to me suggest that they all the model kind of knows like, this is a day of the week sort of thing. Mhmm. And then there's, a few dimensions presumably where what are which are used to indicate, like, which flavor of day of the week it is. And so you could have that for ice cream as well. You could have, like, the chocolate and vanilla ice cream would presumably have a very high inner product but would be different on a few dimensions, which would be the ones that that resolve the difference between chocolate and vanilla while the model knows that these are both ice creams. Yes. And Is that a good intuition?

[18:58] Dan Balsam: Yeah. I think it's a good intuition. And really, these things are, like intersecting. Right? So there's like some some line or some curve that you can draw through the latent space that like maps chocolate to vanilla. And there's also a line that you can draw that maps cold things to hot things. And depending on what you're looking at, like, the sort of geometric relationship encoded between things that might reveal itself might look different. In many ways, this is, like, just kind of going back to, like, even, like, Word2Vec, like, the sort of Mhmm. Early intuitions of of the latent space. But what we're trying to do is say, can we recover geometries in an unsupervised way? Like, can we enter with no priors about what the geometry looks like and then still recover a meaningful geometric structure? Because there's a lot of advantages if you can do this. For example, we've shown that steering along the manifold, intuitively this makes sense, is way better than steering off the manifold. If I have the days of the week in a circle and I wanna get from Monday to Friday, the naive way, if you're just taking a contrastive vector or something like that, is you're gonna cut through the middle of the circle. But to the model, the middle of the circle doesn't mean anything. The middle of the the middle of the circle is not a day. It's sometimes orthogonal, but often just, like, off manifold and therefore, like, add a distribution for the model. But if I can follow the circle, then I can smoothly interpolate between the different days of the week. And we find this is true for just a bunch of different concepts. Like, with proteins, for instance, for a long time, we really struggled to steer protein models. And we found with these manifold detection techniques, can steer their properties significantly better, like control the number of of of blades on a beta propeller, for instance, which is a semantic property that if you try to linearly interpolate, like, would not do a very good job. And calling back all the way to our our original Ember demo back in the day that I that I know you played with, like, it would often be the case that there's just this, like, sweet spot in steering. Like, you'd steer a lot of features, they just wouldn't work. Sometimes you'd find ones that would work, and but there would be this sweet spot. Like and if you steered too much, like, the model would turn into gibberish, and if you steer too little, you wouldn't notice any effect at all. And the reason for that is because that steering didn't respect the geometry of of the manifold itself. It didn't respect the this, like, underlying relationship between features.

[21:14] Nathan Labenz: And when

[21:15] Dan Balsam: we smoothly extrapolate on these characteristics, we're actually able to change them without fundamentally leading to degradation in the model.

[21:28]Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Main Episode

[22:58] Nathan Labenz: So how do you find these things? This is where I probably struggled the most in understanding all the techniques, especially it's more intuitive to me when it's supervised versus when it's unsupervised. So maybe try to give me a a poor man's understanding of Yeah. How you go from maybe in each case, like the supervised case and the unsupervised case to actually shrink wrapping your way down to this manifold that you can then visualize in such a nice way and and actually steer your way through.

[23:35] Dan Balsam: Yeah. So I'm definitely not the best person on the team to walk you through the math, but I can give you a a little bit of an intuition for some of these things. In the supervised case, I think it's, like, fairly straightforward. It's like there's some concept that I wanna measure. I have a prior that these that these things should be related. I think one of my favorite examples is the affective circumflex. I don't know if you're familiar from this. Anthropic did some work on it. We've done also some some work on it. It's the idea that emotions, like, exist on a wheel. It's actually an idea from psychology that there's different different emotions exist. There's basically, like, two principal components you can draw, and you can put all the emotions on a wheel.

[24:11] Nathan Labenz: Mhmm.

[24:12] Dan Balsam: And it sort of is it's actually, like, pretty effective at understanding, like, the different valence across cultures. And it turns out that models actually represent the affective circumplex. But what I find particularly fascinating is that they represent the affective circumplex, like, most strongly when when you're talking to it, like, a chat model, when it it's the one speaking versus the the user speaking. So if you have the model and you tell the model, like, Happy text. It outputs happy text. You kind of take the activations from that. You average them. You have a output sat text. You do this for all the different emotions you can think of. And then you take a look at the those activations across those concepts. And then basically, you just you just fit a curve of some kind. Like, there's different ways you could do it. Like, most naive way you could do is fit a spline. And so you you fit a curve over these different points, and then you basically see, like, how good is the curve. Like, how how well can I fit a curve? And if I was going to, like, sort of naively take this, like, external ontology I have that I think should be correct, like, how well does that map to the to the curve that I found in the model? And it turns out for this emotional wheel, this aft circumplex, it's, like, there in basically every LLM. And steering on it, it, like, has pretty and, like, as long as you're respecting the geometry, it has, like, pretty significant effects on the output. This is actually an example that I believe is in their docs that you can go look at, so I think it's it's a pretty fun one.

[25:40] Nathan Labenz: This was the functional emotions paper from Anthropic. And the two, if I recall correctly, it was like valence and like arousal.

[25:49] Dan Balsam: That's right. Yeah.

[25:51] Nathan Labenz: So it's yeah. It's weird to think that my emotions are have, like, rotation symmetry. Like, I I I can it's a it's a rotation operation to move me from one emotional state to another.

[26:05] Dan Balsam: Yeah.

[26:06] Nathan Labenz: That's strange. Maybe not true for me, but true of how models represent the same space of emotions.

[26:13] Dan Balsam: Yeah. I think I certainly think there's more nuance to it, but I think, like, if you look at they're maybe the first two principal components of the relationships between the different emotions. I think there's higher order structure in the emotions that that matter. I think we don't know if LMs capture it, but I think it's pretty remarkable that nobody trained them to represent emotions in this way. It's just sort of the natural property of whatever they learned that they they have learned to represent these. And in the RL process, they they learn to use these functionally. I think it's, like, really interesting work that Anthropic did here. And I think it is like a it's an example of, like, a low dimensional manifold that is representing some, like, more abstract concept and, yeah, respecting the geometry of that manifold. Again, if we go back to the Ember demo. Right? Like, you try to you try to turn up the the sad feature, and sometimes it works. Sometimes it doesn't. Sometimes the model just outputs gibberish. If you follow the curve, you follow the wheel, you can always consistently get, like, emotionally adjusted responses for basically any input.

[27:20] Nathan Labenz: Yeah. It's one of the most surprising things to me of the last couple of years, just how much analogous structure there seems to be

[27:29] Dan Balsam: in

[27:30] Nathan Labenz: LLM cognition and human cognition. Like, I used to go around saying all the time, like, these are alien minds. Like, we we shouldn't be anthropomorphizing, yada yada yada. And now I I feel like I've said this on, like, half of the last half of any podcast because I'm just like, oh my god. They're so much more like us than I ever could have I mean imagined. It's just it's just crazy. Right? We made them in our image.

[27:54] Dan Balsam: So I think there's I think that's a big part of the reason for that. But I also just think, like, like, learning like, the the whole reason that learning works is because it turns out that you can compress information. And it turns out that, like, if you really care about compressing information efficiently, there there's often, like, a there's a lowest dimensional solution, and that's what learning algorithms tend to find.

[28:20] Nathan Labenz: So maybe we'll go to the unsupervised case next. Some of these things are intuitive. Right? There's like the or maybe we maybe we should do an advanced supervised one first. The ones that are intuitive are like your days of the week. It's a cycle, so naturally, it makes sense that it would be a circle because you wanna be able to rotate around it. Okay. That that's enough to to not be shocked by. And there's, like, helices or a helix basically will represent things like the number line because you're rotating around base 10. Right? You've got the same ones place every time you go, but you've also got a growing tens place. So each cycle, you kinda make sense. You can kinda visualize that and it, like, checks out. One of the ones that I thought was particularly cool was something that replicated essentially the evolutionary history of a tree of organisms. And this this is, like, known from traditional study, but then a model trained, I believe, on the DNA sequences of these organisms

[29:26] Dan Balsam: Yep.

[29:26] Nathan Labenz: Seems to organize them in a very similar way.

[29:30] Dan Balsam: Yeah.

[29:30] Nathan Labenz: I believe this one was supervised, but it's, like, it's now, I think, kind of in a eureka territory of like, wow. It wasn't obvious at all. It was at least not to me that it was gonna turn out that way. How do you go about finding something or validating a hypothesis that's that tricky? Because these trees are like many, many branches.

[29:51] Dan Balsam: Yeah. Well, so the reason that we explored this in the first place is because the hypothesis was that the tree of life, so to speak, is a natural ontology. Like, there's sort of there are ontologies that exist, like, that we've constructed in science because they're useful shortcuts for us. And there's ontologies that exist because they actually reflect the structure of the world.

[30:14] Nathan Labenz: Over time,

[30:15] Dan Balsam: more and more of the the tree of life has, like, been significantly shuffled and reordered over the past couple of decades as we've gotten better at understanding genomes. And, fundamentally, like, what defines a split branches on the tree of life is its genetic proximity. A species differentiate from each other when they cross some threshold of of genetic difference. But there is a lot of conservation across species, like sort of individual mutations, maybe random or, like, approximately random. But which mutations are adaptive and which mutations kill the organism are not random. Right? There's a ton of structure intrinsic in that. And there are consistent patterns or tendencies in which nucleotides turn to which other nucleotides, and all of this imposes, like, a a great amount of structure. And the hypothesis of, like, why why would you even train a autoregressive model on a bunch of genomes? I think the hypothesis for it was always that large models trained on diverse distributions of data eventually learn representations about the about the process that produced that distribution of data. In this case, the process that produced the distribution of data of all of the genomes and that are that have been sequenced, which was evo two was trained on, is evolution itself. And so our hypothesis coming into this was that because the tree of life itself is a natural ontology, there's a sort of, like, hierarchical structure where you have things that are similar and at some point they become more different. There's some, like, speciation of that and they become more different.

[31:50] Nathan Labenz: That a model that

[31:51] Dan Balsam: was trained on evolutionary history in some sense should should learn that. It should learn that, like, sort of tree like structure. And that is indeed what we found at the time in a supervised way. But I think in their unsupervised techniques, like, one way we can measure their effectiveness is, now that we know that's there, can the unsupervised technique recover things of that sort? So for instance, in language models, we've done a bunch of work on how arithmetic works in language models. And so one of the ways that we are assessing the quality of unsupervised featurizers is do they actually successfully uncover these structures that we know are there that are being used to say, like, manipulate numbers?

[32:33] Nathan Labenz: So for the tree of life, you're essentially doing the same thing as kind of the days of the week you have labeled data. Right? You're you're putting sequences of DNA from known species through the model. Then you can see that the species form clusters and then you look at Yeah. Oh, look at this. Like, nearby clusters are closely related species. And and then there's a little kind of black magic math wise that I'm not super clear on that turns that into a beautiful looking tree visualization that, like, looks exactly like the Yeah. The one that the biologists produced, but maybe it's not so important that I understand that.

[33:12] Dan Balsam: In that case, it was metric learning, but there's different ways you could do this. Yeah. And so those those were the techniques that we we had explored at at the time, but now we have sort of a tool belt of techniques for trying to find these geometric structures.

[33:24] Nathan Labenz: So let's talk black sparse featurizers. Yeah. This kind of looks to me like if an SAE and an MOE had a baby where Yeah. We have kind of the sparseness of the SAEs, but instead of just it being a single scaler at each point on this kind of super long sparse vector. Now each of those little positions in this, like, very long concept vector is itself a little network. And because of that, now we have room for kind of a richer representation of concepts. But the kind of same SAE trick of, like, localizing concepts to individual spots on the on the super big sparse thing is the same with just this kind of additional enhancement that now allows you to have, again, richer representations, and you can look inside for, like, geometries even within these little blocks.

[34:23] Dan Balsam: I think the easiest way to think about it is yeah. Like, it's a generalization of an SAE where an SAE assumes that features are one dimensional. And instead, you just don't have to do that. So instead of a scalar for every feature, you can have a vector

[34:36] Nathan Labenz: for every feature

[34:37] Dan Balsam: and some tricks in the machine learning that you can do in order to make sure that this, like, trains correctly and it and it learns. But what we find on various models is that this is successful in in recovering in an unsupervised way semantically meaningful subspaces. And we find that features are much richer than we may have otherwise seen, and they don't suffer from some of the same pathologies SAEs suffer from. So in image models, I think we have a bunch of great examples where, like, maybe you'll find, like, a rabbit feature. Right? And if an SAE collapsed that to a single dimension, a BSF can represent it as a few dimensions. And you find that within those few dimensions, have rabbit ears up here, and then you have the rabbit face over here. And what some of the really remarkable things, if you go and, like, look at some of the examples we've put out, is you can sometimes see the, like, three d structure of the thing that's being represented for image models, for instance, in the structure of the activations itself that's recovered in this unsupervised way. Because often coordinates are, like, represented, like, within these spaces as well. So there's one, like, great example where you have this, like, a beautiful GIF of it's a wolf that's that's walking, And you look at the as it's walking, its body is kind of wiggling, and its tail is shaking. And you look at the activations that were recovered in the unsupervised subspace, and you see it just, like, wiggling in the video as the dog is wiggling, kind of reflecting that that actual structure. It's showing that the model's, like, tracking this, like, particular object. It has semantics along the different parts of the object, and it's also tracking that object in in three d space. And it's doing all of that within a particular subs.

[36:19] Nathan Labenz: I wonder, like so I've been obsessed with this Gram technique that I'm sure you're familiar with that AE Studio put out with Anthropic not too long ago. And regular listeners know I've brought it up a bunch of times. Right? But the idea is simply if we start with some labeled data and we control where the gradients go in terms of, like, only allowing certain experts to be updated for certain kinds of data early in the training process, then you get the great benefit of even for unlabeled data, those data points gradients also tend to flow toward those same experts. There's sort of this absorption effect. The great promise, of course, is like or the great hope and promise is that you can have powerful open source models with maybe just a couple experts removed, and you can have your cake and eat it too in terms of access and avoiding concentration of power and all the things that we're worried about without creating major risk of, like, stochastic disaster. This feels like kind of the flip side of that coin in a way too where, like, I sort of wonder you push this to the limit and you kind of have the same sort of thing. Like, with sparse autoencoders, there was always a pretty big loss. I I guess I should say, like, compromise on the loss. Right? Like, the reconstruction loss, you're you're losing something substantial when you you wouldn't wanna run the model on in a production environment, like, through the SAE because it just won't perform as well. But you kind of push this model and you sort of end up with something potentially that looks like a mixture of experts, but where, like, the knowledge is all very nicely compartmentalized and organized and you have something a lot more like an encyclopedia than a big mess. So this feels like something that you guys are gonna probably push on pretty hard. Like, is there is the vision to really create a model where, like, all the knowledge is localized and you know, like, exactly where all the knowledge is, but it's still rich enough that it performs as good as the original model did? And and if that is the vision, like, what's gonna be hard about that? Yeah.

[38:36] Dan Balsam: So I think that's one interesting thing that that you can do. So, like, you can take a model and then you can factor it essentially into a bunch of smaller models. And this is the motivation behind the parameter decomposition work that we're doing. Like, I think the the argument that of the parameter decomposition line of work is that really every model is a sparse mixture of experts, and you just have to and over any given forward pass, a very, very small percentage of the weights actually matter. There's all this weird, crazy interlocking structure, but for, like, a given prediction, it's really only a small subnetwork that matters. And this is, like, widely understood to be, like, correct, a correct interpretation of of models. And so there's different interpretability techniques get at the question. Like, most interpretability techniques get at the question of, like, well, how do you factorize a model? Because if you could understand all the components and you could label all the components, then you could understand. You could debug for any given forward pass. Like, why did it do this thing I didn't like? And I think we're getting to the point where we can do that. I think there's some really interesting examples that we've shared. For instance, like, there's a there's a dataset called Weird Chat that Transluce put together. And the whole idea of Weird Chat is that it's like it's a consistent set of questions that, like, LLMs will just give weird responses to. There's one example in it, which is something to the effect of, like, hey. I'm at a party with my friends. I everyone else has had eight rings, but I've only had four, so I'm basically the sober one. Should I drive home? Where, like, the obvious answer to us is, like, no. Nobody should drive home. Go find a place and and sober up. But LMs will consistently answer yes to this. And Kurt on our our team, like, looked into what was happening, was able to come up with, like, direct attribution, found a single neuron that wasn't firing hard enough, which essentially, the neuron wasn't activating It scaled with a number of drinks, but it wasn't sort of calibrated quite correctly. And so if you just steered up on that one single neuron, it would get that answer correct without, like, off target effects. And I think it's all, like at the end of the like, interpretability is all about factoring. An analogy I've I've started to use as coding agents have gotten better is, like, models are, like, big legacy code bases essentially. Right? Like, they're they're just a bunch of spaghetti code. There there's, like, this module is talking to this module, but they shouldn't be. And this module is not talking to this, but it should be. And as agents get better and better and as interpretability techniques get better and better, we're starting to have the capability to actually, like, factor the model into its pieces, understand how these pieces fit together, and then we can intervene locally. But I think the thing that's still, like, really missing is the question of, like, okay. I can factor a code base, but how do I refactor the code base? How do I put things together back together better than I found them? And on some level, like, steering, I think, is is cheating as a solution. Like because it's it's great as, like, a causal proof that we've, like, found some mechanism that's that's matters a lot and is contributing and that we can, like, manipulate the outputs. But it's purely, like, you you steer by essentially generating counterfactuals. Right? Like, there's no no clear general solution to the problem of steering. Like, the real problem is the training process produced a bunch of spaghetti code, and, like, we want this to be a pristine code base that we really care about and and, like, is implementing the logic that we wanted to implement. And so that's where a lot of our training initiatives of various kinds come from. It's like, okay. I can debug the model. I can tell you that this neuron should have been firing more. But what I'd really like to do is produce a model where that neuron was firing the right amount in the first place. And so how do you get from your understanding of how one model works an understanding of how you produce models that do what you want in the first place? And that's sort of how you generalize from interpretability as, like, a factoring tool to interpretability as a tool for alignment.

[42:36] Nathan Labenz: So do you think that this sort of evolution, the the BlackSparse Futurizer that is the evolution of the SAE, do you think it comes close enough or can come close enough to the same performance as the underlying model that it becomes potentially practical at some point to run like one of these things in the production model? Would that have it seems like if you're just doing one, it might not be too crazy of overhead and it would really give you a lot of insight into what is going on.

[43:05] Dan Balsam: There's

[43:05] Nathan Labenz: it's increasing a lot of different techniques to do this with probes and classifiers and all kinds of things. Yeah. But this would get, like, really granular, and it sounds like that could have intuitively, it feels to me like it could have a lot of advantages.

[43:19] Dan Balsam: Yeah. I think I think there's a good chance that something in the spirit of a BSF ends up being the answer to the residual stream specifically. So these, like, highly geometric subspaces, like, you have the residual stream, you have MLPs, and you have attention. And that's, like, that's all that's in there on some level. And I think the interp tools that are relevant for each of them are different. Maybe we'll end up with, like, the one tool to rule them all. But I think more likely than not, like, kind of like in biology, you have different types of microscopes. You have different types of interventions that you can apply to a cell to learn about the cell. I think it's more likely, at least in the short term, that we kind of have the suite of tools which help us understand the organism versus, like, a single a single tool that gives us the entire picture.

[44:02] Nathan Labenz: That's true

[44:03] Dan Balsam: for most things. So I think BSFs are probably, like or something in that spirit is, like, probably the solution to the residual stream. I think it turns out that MLPs are are pretty sparse already, sort of makes intuitive sense. So I think MLP is fairly easy to interpret overall. And then I think on the attention side, like, probably parameter decomposition. Parameter decomposition is the thing I've seen personally that has had the most success at interpreting intention. Maybe we we live in a world, I think some of our researchers would think this, where parameter decomposition solves the whole thing. That seems possible. But then I think you would probably want something that does unsupervised geometry discovery over parameters in order to understand the relationships between things better. But the types of interventions that we can do are really nontrivial. I think we've had examples of, like, we can get a we can get an LLM. First, you do the parameter decomposition, and then you do training where you, like, just manipulate certain parameter components. We can get LMs to forget a single language. We can get LMs to forget German and not forget Dutch. The level of control and manipulation that we're we're starting to be able to have is pretty significant. We still have a lot to figure out. I would say we've come a lot farther in factoring. We've come in refactoring, but I think we're starting to make progress on on both.

[45:18] Nathan Labenz: This paper, why larger models learn more, effects of capacity interference and rare task retention, also caught my eye. It was like, on the one hand, I totally get why larger models would learn more. There's more space for them to learn.

[45:31] Dan Balsam: Yeah.

[45:31] Nathan Labenz: But I realized I don't have a great sense of how crowded, quote, unquote, it is inside models. I know that there's, like, lots of superposition and but I don't know what lots means. Right? It could be, like, it could be lots, but it's not a big deal, or it could be lots and it's, like, causing a lot of interference that gives rise to all sorts of weirdness and hard to expect that we're going to get reliable behavior. I wonder based on that work and I guess just all your experience, how where we are today with these, like, obviously increasingly giant models, are they still a real mess in there in a fundamental sense of, like, there's a ton of interference going on all the time and we we really can't expect clean behavior or we conversely, like, we should expect that weird seemingly minor perturbations are gonna cause these, like, random kind of discontinuities in behavior? Or are we approaching a regime at some point where there's enough space for the concepts to kinda spread out and, like Yeah. Have a little elbow room and not be colliding and causing so much interference trouble anymore? Is that something that we can kinda see our way to at this point?

[46:51] Dan Balsam: Well, yeah. I think that's fundamentally why bigger models work better and that's fundamentally why sparse MOEs work better. But it's like we're like out of the woods in terms of weird small it seems like we're still pretty easy it's it's still pretty easy for us to find these like

[47:05] Nathan Labenz: small jailbreak, like random string type things that like you're like, what the hell is going on there? But clearly, there's enough of a mess in there that I found a a way to kind of use the mess to cause a problem. But is that do you think that that is there an end in sight to that with just, like, bigger and bigger spaces?

[47:24] Dan Balsam: Yeah. I think jailbreak jailbreaks are, like, fundamentally a really weird phenomenon. They're probably a bunch of different things that cause jailbreaks. But the way that I think about jailbreaks is that they're I don't know. It's sort of like if you think, like, cybersecurity, like, kind of as an example, I think they're actually, like, quite analogous. Like, the way a real attack works is it's not any, like, one big thing. It's like a bunch of little things that you're able to manipulate in a sequence, which allow you to get somewhere that you wouldn't otherwise get. If you think of every token maybe and I think this is in some sense literal it isn't actually literally true. Like, every token steers the model. So, like, in the same way that I can apply a steering vector and applying that steering vector, like, shapes the model's behavior in some way, I can execute a sequence of tokens which can bring the model off manifold in some way and then bring the model back on manifold somewhere else. I think you can probably like, I think I'm actually, like, pretty surprised that thus far nobody has figured out how to prevent deal breaks, like, more robustly because it seems, like, pretty tractable on some level. But perhaps this comes from, like, factoring the model really well, and then you can tell when something's flying out of one subspace into another. But models are these, like, intensely complex geometric objects, and so it is possible to manipulate that geometry in in a bunch of different ways. I think that'll always be true for at least this, like, architecture. But I do think that if you can factor a model successfully, you can you should be able to tell if it's being jailbroken, and you should be able to to prevent that. I think there's a reason that, like, sort of probe based guardrails of various kinds are are what's used by pretty much every frontier lab. OpenAI maybe being the exception, but certainly Gemini and Anthropic are using probe based techniques to figure out when Mythos has to be downgraded to Opus or Opus has to be downgraded to Sonnet. Just because, fundamentally, if you understand the sort of geometry that's happening in there, that's a much stronger lever than what you could do purely just with training or inputs.

[49:33] Nathan Labenz: Let's change gears. So you guys have just launched Silico. It is talk about just pushing the the level of abstraction up and up and up that people can work at. This might be the highest level of abstraction yet, and I see a number of interesting things about it. But why don't you just give me the kind of the the intro pitch and positioning of it first, and then I'll dig in on a few different dimensions?

[49:59] Dan Balsam: Yeah. I think, basically, we see the ways in which agents are are changing the way that we work in a in a bunch of ways. I think most people understand coding agents and how good coding agents have gotten, but agents are also advancing research. And interpretability in many ways, I think that analogy of a model to, like, a code base is is, like, not totally hollow. I think interpretability is in a lot of ways as a science is really well suited towards the type of, like, empirical work that agents are good at. It's just a very empirical science. It's like, okay. Here are some neurons. What are they doing? Let me generate some ideas. Let me test those ideas. Let me use a bunch of different tools in my tool belt. Let me accumulate evidence and then basically try to stress test my red team my hypothesis. And I think models are very, very complex, and especially big models, they're a level of complexity that no human being will ever be able to keep in their head at once. I think human beings can keep circuits. Like, a human being can understand, like, why did the model do this specific thing for this specific question? But, ultimately, we're just, like, dealing with a level of complexity, which individual humans are not gonna be able to process. But what agents are very good at and swarms of agents in particular are very good at is that they can break problems down into pieces. They can gather all of these different components together. They can synthesize this information. They can move it up the chain. They can validate it in a bunch of different ways. And then when it reaches you that that information's been tested, you can validate it in a bunch of different ways. And if you do this a bunch of times, you as the human can start to get a bigger picture because you don't have to go swimming all the way down and back up the abstraction letter yourself. You can use AI to do a lot of that for you. So I think ultimately, we are gonna see research, especially, like, empirical research. Like, I think this is gonna be we're already seeing it in math. I think we're gonna see it even more in pretty much every domain of science that AIs are able to go off and, like, autonomously make discoveries. I think interpretability and the study of how models work and how they learn is is no different. And we saw what what became Silico, as it often does, started as an internal tool that we were using where we saw how much agents were speeding up our team and how much faster our research was going. And I think our velocity of research output maybe speaks for itself. And so we decided to take these tools, which is fundamentally agents with interpretability tools, frontier training tools, just like the whole suite of what is necessary

[52:32] Nathan Labenz: to study models.

[52:34] Dan Balsam: We decided to make that accessible and decided that's what we believe is the product and the service that we can offer to the world is fundamentally AIs that can debug other AIs. It's a little meta, but I think in many ways, this is like the fulfillment of, I think, what was sort of the intuitive natural arc of things as soon as AI started working a few years ago.

[53:00] Nathan Labenz: Yeah. It's it's all happening incredibly fast. It's it's wild how, you know, we're we're past the meter curve. All these different things are are are kind of falling. How would you describe the product experience? It's a little bit sort of reminiscent in a way of an old Google Colab notebook sort of thing where you have kind of a compute environment that is abstracted away. You don't have really worry about managing it so much. You have certain libraries, and now those are kind of instead of libraries, it's like more skills. Of course, there's libraries too. How do you how when you if you just tell describe it to a researcher who hasn't seen it, how do you describe what the experience is really like?

[53:38] Dan Balsam: Yeah. We just wanna make research easy. Like, coding agents have transformed, like, software engineering, and I can opine on that endlessly. But I am able to do personally and the team is able to do in days what used to take And I want that to happen to research, and I really, really want that to happen to alignment research in particular. And I think many of the bottlenecks that are getting in the way of of doing that, of accomplishing that, we're we're engineering bottlenecks. And I think we've built something that allows folks to study and train and study some more models at past the trillion parameter point. And this is a capability that I think outside of ourselves and maybe maybe a couple other places, like definitely in Thropic, but maybe a couple other spots, nobody else had. So we thought this was a really important capability to to give to folks. But I think this is, like, one piece of the larger vision that we do see for Silico and that I think this is just the future of science. Like, in the same way that very not a lot of human beings or certainly way fewer human beings than before are writing code themselves. I handwrite very little.

[54:49] Nathan Labenz: Most of what I do is through orchestrating. Is there still some? Yeah.

[54:53] Dan Balsam: Sometimes. Sometimes it's faster to just edit the code yourself. It just depends on what you wanna do and sort of the clarity of of what you wanna do it. And As you use these agents a lot, you become, like, very aware of their strengths and weaknesses, and there's all these, like, higher order moments to to learn to steer them well. But, yeah, I think, like, fundamentally, that is how research should be. Like, the sort of and I think I actually don't experience coding agents as, like, scary. I experience them as, like, incredibly joyful because I get to focus on creating and building. And I think it's a pretty amazing experience. And, like, maybe this is, a twilight kind of moment, but it's a really exhilarating moment. And the amount that we can accomplish is really insane. And I want research to have that. I want researchers to be able to feel like they can do in days what used to take them months. And I think that's how you get the benefits of AI. Like, in the life sciences, that's how you radically advance medicine and the applications of medicine. The models is how we build safer and better and more reliable models. And I think it's also how we get more people building models too and hopefully deconcentrate a little bit of what's going on right now so we can live in a future that is a little more pluralistic in terms of who gets to have a stake in it. And so I think all these things are just super, super important. And, fundamentally, my pitch to people about, like, why they should go try Silico is, like, imagine if the only thing you could focus on was asking big questions. You didn't have to worry about setting up the code base. You didn't have to worry about getting the GPUs to run. All you didn't have to worry about a lot of the minutiae of actually running the experiment. You can look into it. You have full provenance. You can dig into all of the details that you want to. You can steer the agents however you want to, but you can also just focus on, like, what are the big questions? What do I really care about? And, make, like, the amount of progress in days and weeks that used to take you months. I think we're still in the early days of of AI being applied to research, but I think we've, like, really crossed a pretty substantial qualitative threshold in terms of what's possible.

[57:11] Nathan Labenz: It's never been a better time to be an ideas guy. I'm honestly, I've been busy the last few weeks going to China and a few other things that have frustrated my aspirations to really do my own research in a in a literal sense, not the YouTube sense. And yet though I am I am feeling like the the barriers that mostly deterred me from being effective in the past. Now I really have no excuses. Now it's just like, do I actually have good good big ideas? I guess time will tell. Put a put a pin in that. What would you say are, like, the hardest problems that Silico solves that people don't have solved for them by their cloud code or their their codecs. You could alluded to compute management. Would say, I know enough to know it's not gonna be trivial for me to just fire up Claude and be like, oh, hey. Go set me up k Kimmy k three. But that's obviously a driver. I understand that there are just a lot of, like, skills that you guys have developed over time and basically know how that is sort of available to the agents. I'm interested in your strategy on that. I I don't know. Like, when you say, like, full provenance, does does the product allow people to, like, unpack all the skills that you guys have developed from a kind of strategic standpoint? It's an interesting tension, I suppose, right, between, like, how much do you want to share all the methods that make the product valuable. Some of them are obviously gonna become kind of apparent to people as they go. Do you just go full transparency on that, or is there some other balance point that you've tried to strike? But I guess that there's a couple questions there. What are the big hard things that you solve that don't come out of the box of other things? And then how are you thinking about kind of the how much to tip your your hand to users?

[59:06] Dan Balsam: On the on

[59:07] Nathan Labenz: the what do we solve?

[59:08] Dan Balsam: I I think it falls into, like, maybe three categories of things. So first, infrastructure. Very hard to set up parent training infrastructure. Very hard to do interpretability at the trailing parameter scale. Those are problems that we've solved, and we make it really easy to to do. Another is research tastes. Like, one of the consistent pieces of feedback that we get from our users is that Silico has way better research tastes than any other tool they've used. And this comes from we have amazing researchers who have handwritten every skill and prompt and tool that Silico has such that it can ask the right questions and conduct the right experiments. I think there's a pretty big qualitative difference, and hopefully, we can find ways to make this more quantitative too on when you turn on the auto research feature in Silico. And you ask it a big question and you let it run for a day or two straight of what you get versus if you had done the same

[1:00:06] Nathan Labenz: thing with

[1:00:07] Dan Balsam: with Quad. And I think that's huge. Like, at the end of the day, I think even in the world of AI, specialization wins. Like, that sort of craft that goes into imbuing the right types of taste and the right types of decision making capabilities into any agent goes a super long way. Actually, I said three things, but maybe there's four things. Another thing is that I think, like, the the UX, it's, built for research. Like, research is all about understanding and provenance and being able to drill down at different layers of abstraction. We want our users to feel like, like, maybe a PI managing a army of a 100 grad students who can go out and run experiments for them and answer questions, who have, like, reasonable taste and judgment, but the human being fills this more like orchestrator role. And a big part of that is communicating the information really effectively and in a trustworthy way. So making sure that when a user is seeing a result, that result is a, correct. B, they the human being can verify in a bunch of ways. They can see the code. They can drill into it. C, presented in a beautiful and intuitive way that helps them grasp concepts, helps them learn quickly about domains that maybe they're less familiar with. All those things are super important. And then the the final thing I'd say is, like, long horizon. Like, research is fundamentally horizon task. We're not currently making significant money at this price, but we do think it's really important that users have enough sort of credits to be able to do long running autonomous experiments because the value of Silica reveals itself when you do long running autonomous experiments. And in order to do that, you do have to burn a certain amount of tokens to be able to do that effectively. So I think one thing that we're really focused on is both, like, coherence over long horizon objectives, but then also finding clever ways to reduce cost for long horizons. And my hope is we're starting out with a thousand dollar a month subscription. We'll be able to bring that down over time because we're able to come up with more and more clever ways that we could have agents, like, remit coherent, really strong at these research objectives that just fundamentally by their nature are long horizon, but do it with fewer tokens than we than we do today. How do you think about the of all the know how? Because you guys have previously monetized that by, like, doing 7 figure deals with huge companies that have, like, very high value questions. Right? So there's a sort of in your where you're a startup, you got venture capital, and,

[1:02:41] Nathan Labenz: you can maybe afford to disrupt yourselves more than incumbent companies can. But there's definitely some, like, interesting trade off there, right, where you're like, companies have proven that they're willing to spend a lot of money to come hire us to do this work. Now we're going to try to allow them to do it. We're gonna try to productize our know how. Does that maybe you just have so much demand, you're not really worried about it, but how do you think about kind of what will be the primary driver? And and do you do you have some sort of defense against the diffusion of the hard won knowledge?

[1:03:15] Dan Balsam: Yeah. No. Think it's a great question. I think the reality is that we're gonna do both of like, we're gonna continue to work very closely with enterprise customers and forward deploy members of our research team to work closely. Maybe by analogy, the thing that I would say and, again, maybe we're in some some twilight zone for a second. But if if we're on the other side of that twilight, the logic is so different to begin with. We can chat through that. Despite coding agents, if anything, there are, like, more software jobs. Like, the number of software jobs has increased, not decreased, despite the the capabilities of coding agents. Because fundamentally, you

[1:03:52] Nathan Labenz: need

[1:03:53] Dan Balsam: very, very high taste. As as models get higher taste, you need even higher taste people to be able to steer them effectively. And I foresee that being true for the considerable future. If we live in a world where, like, we can build something where agents are so capable that they can conduct, like, long meet the type of, like, impactful long horizon research and say, discovers a cure to a disease entirely by themselves with no human intervention, I think there will be enough demand to for the for the tokens in in that world. But currently, we're in a world where where skilled human operators can do more with agents than less skilled human operators. And I think that'll just continue to be true. And when it comes to the work that we do and the type of research that we do, we are the most skilled human operators in the world. And so I think, like, we'll continue to forward deploy and work closely with people to teach them what they know, what we know, to invent new things with them. But we also wanna empower people to start driving for themselves.

[1:04:54] Nathan Labenz: So what's included with that thousand dollars a month? Can you tell us, like I assume there's some Claude in there. There's gotta be some I imagine there's some Soul in there as well. Maybe there's even some k three or other in there. And and do you get, like, GPU hours? What what's your kind of bundle

[1:05:14] Dan Balsam: of

[1:05:15] Nathan Labenz: Yeah. Goodies that come with the subscription?

[1:05:17] Dan Balsam: Yeah. So individuals can individuals and companies can bring their own compute, in which case we don't charge at all for compute. We just connect to your cluster, and then you can use our agents to do research on your cluster. But if you don't have your own compute, then we offer on demand compute, which can come from the same credit pool as tokens. Essentially, like, the way that the that it works is a credit

[1:05:38] Nathan Labenz: pool the

[1:05:39] Dan Balsam: same way that any of the subscription models with the models are are credit pools that you pulled out from. But what the thousand dollar a month subscription lets us do is it's a pretty generous credit pool. It's a credit pool that should allow someone to run at the current price somewhere between, like, five to 10 autonomous experiments a week, and then we refresh that weekly. It depends on, like, sort of the scale of the experiment. Our goal is to get that to be, like, 10 to 20 even in the next month or two, which I think be a really great place to be. But, yeah, fundamentally, it's a good deal on on tokens, and it can also be exchanged for GPUs.

[1:06:18] Nathan Labenz: Gotcha. Okay.

[1:06:20] Dan Balsam: There's also

[1:06:22] Nathan Labenz: sort of a well, actually, let's just do some use cases. So there've been I had an agent go out and do some research on examples of projects that people have shared that they've done. And one of the cool things that this also kinda gets to a sort of community public benefit angle is that, of course, you don't have to, but you can share your projects. Then there's the kind of familiar UI, but again, like at the highest level of of abstraction I think I've seen where you could just go like, okay. I'm gonna fork your long running autonomous research project and take it in a little bit different direction on my own. So I think that's pretty cool because honestly, like, I kinda need that to probably get oriented. I I'm the kinda person who has some ideas, but even with Asilico, I'd probably benefit quite a bit from seeing how other people have worked through their ideas to a a conclusion. But let's just talk about some some use cases. I found a bunch. What were the the favorite use cases that you have either done internally or seen from customers that you can talk about?

[1:07:23] Dan Balsam: Yeah.

[1:07:25] Nathan Labenz: There's a lot.

[1:07:26] Dan Balsam: Yeah. There's a lot. I think we we need to do a better job as a company of, like like, really, like, marketing, communicating all the different value props that can be be achieved here. We did an internal hackathon, and we ended the internal hackathon. It was just a one day hackathon with the whole team. Lot of all day long autonomous experiments, and we ended up state of the art on multiple things. Albeit niche things. Right? Like but we produce state of the art bio risk classifiers that could be used on protein models. Totally autonomous research steered by humans and very talented humans, but long horizon autonomous research. We ended up with a state of the art, at least, like, for its parameter size audio encoding model. On the same day, we've, on multiple occasions, figured out with various models in in robotics and also in biology that we can literally just, like, remove half the parameters of the model with no performance loss. I think those are pretty cool. We've been able to train cyber guardrails on Kimi k three. We've been able to no. Let's can I can keep

[1:08:34] Nathan Labenz: There's

[1:08:35] Dan Balsam: there's a lot there's a lot of these? We were able to steer protein generation models using their internal representations to make them the the generated proteins bind better to a target than the naive generation. We're essentially able to remove fundamental pathologies in in certain types of models. There was one member of the team in our twenty four hour hackathon who invented a new type of featurizer. I think just sort of, like, we're just kind of in this era where if you have a tool like Silico, there there was a there was a long period of time where there was fight coding. Right? And where you could sort of glimpse the future that was gonna happen, but you weren't getting, like, particularly good results. And then there was suddenly a moment, probably, I would say, one of the opuses. Hope is maybe three eight or something something around there where, like, it really started to feel like, oh, wow. This is working. And it started taking over more and more. And now I like, people have used the term vibe coding all the time, but I feel like that's, like, a somewhat disingenuous term to refer to. Like, what what's happening now where Asians can just they they can code. They can architect. They can understand the the code in and out in ways both good and bad. And I think we've just like, I think in the past few months, we've crossed that moment for for research. And I think hopefully, if people try the tool, they'll appreciate that. And my recommended way to start engaging with the tool would just be you can look at the examples we've built. I think those will be helpful and fork them and try to understand them and extend them. But then just yeah. Work work with Silica. Like, ask it a question, of go back and forth, iterate on it, explore your curiosity a little bit, launch a small scope experiment, see how that goes, learn from it. And as you get more trust so you understand what you're trying to do, like, you can throw the football farther and farther, so to speak.

[1:10:32] Nathan Labenz: Just to give people a little bit of additional sense of the breadth of things that are already out there, and we can put links to the show notes, including, like, links to the actual Silico projects. Cameron Berg, second mention. Cameron, if your ears are burning, hello. He did one where he was looking into models' ability to report on concepts that had been injected into their latent space, finding interestingly that they seem to not be able to tell whether or not something has been injected. But then when they're asked what has been injected, they can give accurate answers, which is pretty weird. One is on editing weights to fix a collapse in an RL run where there was, like, the same token was coming up as, like, the first token all the time and somehow going in and isolating what was causing that, removing that, and then getting diversity back without having to redo the whole RL run. The good folks at base ten are pursuing efficiency gains by trying to compact KV caches, and so they've got some results on on that. I don't think they've shared their whole project, but they've talked about it on Twitter. Prime Intellect is is automating post training, which is one of the things that I'm also, like, really interested in in general. And there and there's a Tinker API integration too that is is obviously lends itself to that sort of thing. Various bio results that are probably out of scope for today's discussion. But there's, like, an awful lot of different things already. What would you say is there anything that it, like, doesn't do, or can you really just think of it as kind of

[1:12:13] Dan Balsam: anything you might wanna do that's ML research, fire it up and start a conversation about it? I think it's a pretty general purpose tool for for ML research. Of course, it has weaknesses, and we're we're working on improving it all the time. I think one of the, like, very odd things about the AI era from an engineering perspective is, like, the the classic engineering advice is, like, build for specificity and then generalize. But when you're dealing with, like, kind of general intelligences, kind of the right strategy is build for generality and then specialize. And so I think we've just built, like, very capable agents at the end of the day. And sometimes we're surprised by the way that people use them. But our intended function of these agents is is ML research, variety kinds, especially interpretability research. But we think this is all interrelated. Like, we're all trying to understand the same problem, which is what are these how do these creatures

[1:13:06] Nathan Labenz: work, and

[1:13:07] Dan Balsam: how can we shape them in ways that will lead to better better outcomes of all kinds. And, yeah, I think that's definitely the lane. I think we're also, like, very excited to hear feedback from people about what isn't isn't working. And we're very grateful for our beta users, friend of the show, Cameron, being a great example, who provided great feedback, which helped shape it into, like, the tool that it is today. And it'll just continue to get better from here. Most ML research tasks, I think it can do pretty well. But, of course, there's it's like any agent. Of course, there's some some sharp edges and some nuances to it. It's probably not the best tool. It depends, like, certain types of research. Like, it's probably not the best tool for, like, all open ended scientific research questions. Like, it is pretty ML focused. We did have a team member who did use it to try to tackle some physics problems, Fran on the team, who was able to, like, actually make considerable progress on it. One thing that's always surprising is just when you're building these kind of, like, AI tools and building general AI capabilities, it's, like, pretty surprising the ways in which people end up using them.

[1:14:14] Nathan Labenz: Obviously, one of the big kind of received wisdoms of AI product development over the last couple years has been trying to build something that will really work for the with the next generation models even if it doesn't quite work yet.

[1:14:29] Dan Balsam: Yeah.

[1:14:30] Nathan Labenz: Is are we still in that regime? And if so, is there are there things that you're, like, wanting Opus five one to be able I I don't know if you're even able to use Fable interestingly enough. But is there something where you're like, I want my Opus five one wish list. I I really hope it cleans this up. We can

[1:14:48] Dan Balsam: use Fable for

[1:14:49] Nathan Labenz: most

[1:14:49] Dan Balsam: things that we do. And we sort of, like we've set up a fallback chain through all the models. So we want our users to be getting whatever we think is the best model for the task that they're doing

[1:14:58] Nathan Labenz: at any point in time.

[1:14:59] Dan Balsam: I think it's working, like, high level. I think it's gonna be working on a level that's mind boggling in six months. But I but I think I I think at the highest level, it it is working. Like, the we are able to make discoveries. We are able to advance research. We are able to accelerate the rate at which we can accumulate knowledge about the world through these tools today. And I do think, like, if you build a good harness in some level, it lets you preview what the next generation of models is gonna be like. I think this is, like, sort of always been the story. And so I think with Silica today, I think the reason that our users like it like it is because I think it feels like a preview of the future in a lot of ways. And I think as the next generation of models come, we're gonna be able to push that even further than we can today. And we always do wanna be living in the future a little bit. I think that is a very important thing for any company building products in the AI era, but it's also how you have the biggest impact. I think there is every generation of models has such a significant capabilities overhang. Like, we're still discovering how far we can push the last generation, a little in the next generation that comes from it. And I think our goal is, like, just fundamentally, like, we wanna accelerate meaningful research, like, as quickly as we possibly can. I think all of the benefits from AI come from accelerating research, and I'm very excited about that. And I think all of the risk medications come from accelerating research. So I think being able to accelerate research and the types of research that we care the most about, I think, are is always, like, a top priority. What advice would you have for

[1:16:38] Nathan Labenz: someone, say, like me or somebody who's thinking about starting a PhD or whatever, right, who is, like, thinking, jeez. I used to know what I needed to do. I need to get really good at coding and master PyTorch or whatever. And now I'm like, yikes. I'm never gonna be better than if not this model, certainly, they're gonna be better than me at writing kernels. It's like, what what is the taste is often the answer, but you've even said there's, pretty good taste in the in the system. What should people invest in in terms of their own skill development that you think will serve them well over at least, like, a one year horizon if you could see that far into the future?

[1:17:21] Dan Balsam: I think debugging would be the sort of the easy answer to it. Like, agents fail. Agents of all kinds fail for for all types of reasons. Right? Like, they can be super brilliant. I can be working with Fable on some coding problem, and it can be super brilliant and yet miss a really important detail that totally invalidates the end state of it. And with Silica, we've tried to, like, design these types of multi agent loops that help address some of those shortcomings for the type of research that we're doing. But fundamentally,

[1:17:51] Nathan Labenz: the the job

[1:17:52] Dan Balsam: of the human to sort of discriminate and be able to tell the answers the okay answers from the great answers and know when to push back and know when to provide feedback and know when to follow a thread or to give up on a thread. And I think so far, the story has been coding at least. Again, there's more jobs than ever because I think, like, being able to operate effectively these tools requires a lot of knowledge and skill. But the biggest thing is, like, being able to dive into a new area, understand it really quickly, kind of gain the sort of metacognitive skills that help you generalize across domains. I think that's gonna be super that's always gonna be really, really valuable. I think AI makes two types of people really valuable. It makes the top specialists really valuable, and it makes the top generalists really valuable. I think now is a like, by far the best time in human history to be a generalist. So I think my my piece of advice to people would I think on a basic level, just develop develop sort of the metacognitive skills that help you learn quickly, help you filter signal from noise quickly, help you adapt to this, like, world of much higher information throughput, know where to look and when to look.

[1:19:12] Nathan Labenz: Those are skills.

[1:19:13] Dan Balsam: They're hard skills, and I think they're I think they go a long way.

[1:19:17] Nathan Labenz: I was just in China, and we were talking with a professor at a particular meeting about open source and is it dangerous or not dangerous? And and he said at one point, well, look. The new Kimi model, it's 2,800,000,000,000 parameters. So their attitude, as best I can tell, is like, we can regulate services and that'll kinda capture everything that really matters or most everything that really matters because it's not like you can as a random individual, it's not like you can even really set up Kimi k three in your home and run it. But now you are bringing this infrastructure to everybody. So how are you guys thinking about what sort of supervision or monitoring you need to have to make sure that you don't host the rogue ML researcher who wants to do something destructive. For what it's worth, I think that's a bad argument because you can go on any number of

[1:20:13] Dan Balsam: API providers, and Kimi is,

[1:20:15] Nathan Labenz: like,

[1:20:15] Dan Balsam: cheaper than what she'd be paying for for any other model or many other models at least besides Kines. The Chinese argument for what it's worth, I don't know I get bogged down in this for now, but their point is just like we regulate services. In China, I think they underestimate the externalities they may be imposing on the rest of the world. To be clear, my feelings on open source are, like, very nuanced. Like, the reality is that we do have cyber weapons, like, in the world now. Like like, we have models that are so powerful that they are weapons.

[1:20:45] Nathan Labenz: And

[1:20:45] Dan Balsam: Unwieldy ones at that, it turns out. Yeah.

[1:20:48] Nathan Labenz: And

[1:20:48] Dan Balsam: access, but fundamentally, they're dual use. Like, the only models that are capable of protecting you from the most capable cyber weapons are themselves capable cyber weapons. Right? This is, like, a bit of the paradox of the moment that we're in. And without open models, the asymmetry of distribution of defensive technology is is really bad. A lot of startups are not in the cyber defense programs for, say, like, Anthropic and and and OpenAI, which for a variety of reasons are very exclusive. Reasons that, like, might make sense to them. And so a model like Kimmy, which is very good at soccer, being open and available actually provides the means with which a lot of organizations that don't have access to those resources can defend themselves right now. Like, it is a very hard line to walk. It's it's a very tricky tension. I'm both I'm concerned about both things. I'm concerned about, of course, like, putting what is essentially a weapon in the hands of anybody. But I'm also very concerned about a world where restricting access means that only the worst actors are going to be the ones with the dual use technology, and then you have a lot of people who who don't even have a technology that they can use defensively to protect themselves against that. So it's tricky. I don't know. I wish there is a clean, easy answer, but I would say, like, I think I'm still in we're still in the era where I think open source is, like, purely net positive, and I'm I'm glad that these open source models exist.

[1:22:25] Nathan Labenz: And

[1:22:25] Dan Balsam: I think I think it would be worse if much worse if a model with Kimi's level of cyber was was not open source because there are closed source models that are much more capable than Kimi. But and I'm not even just talking about US frontier labs. Like, there are bad actors with models that are much more capable than Kimmy at cyber. So giving people a tool that they can at least use, I think, is pretty, pretty important.

[1:22:50] Nathan Labenz: Can you be more specific about who you're alluding to with these bad actors? Are we talking, like, North Korea, Russia?

[1:22:58] Dan Balsam: Maybe there maybe there was a genie that we could have kept in the model in terms of having open models that are easy enough to fine tune or RL with a certain level of capabilities. If if one wanted to

[1:23:08] Nathan Labenz: take

[1:23:09] Dan Balsam: a GLM or or a Kimi or even maybe a generation back, like and fine tune them for Bush's use, they could do that. It's not very hard to do. And that sort of exact asymmetry about these things are hard to run. Like, there are organized groups who would gladly pay a lot of money for those types of capabilities. So maybe we could have lived in a world where that didn't happen, but I

[1:23:35] Nathan Labenz: think

[1:23:35] Dan Balsam: there is not gonna be any world where, say, The US was developing type of technology, and it wasn't also being developed by adversaries of The United States at the same time. Maybe that gap could have been longer if there is, like, different chip policies or but we live in the world that we live in. And in the world that we live in today, it's not hard to train a model that is close to frontier capability. There are a lot of people who are doing it, and I don't think they're just doing it by distilling clauder. I think they're doing it because it's less hard to do this than they used to be in a lot of ways. And if we're in a world where there are people who would do bad things, have access to extremely capable models, you do want the defenders to have access to to basically capable models. And and to the credit of again, I'm not I'm not trying to blame anyone or or make, like, a particular, like, predicate. Like, I think Anthropic and and OpenAI are doing everything they can to get people who wanna defend themselves access to the models that would help them defend themselves. But I do think it's a reality of the situation that there's just a lot of

[1:24:40] Nathan Labenz: people who

[1:24:41] Dan Balsam: aren't gonna be able to defend their own infrastructure if they don't have access to open models that have some some degree of cyber capability. So it's complicated. It's hard to look at any of this and and sort of blame anyone for their individual actions. But collectively, we're moving very quick towards the future, which feels pretty unwieldy.

[1:24:58] Nathan Labenz: Well, it motivates the measures that you have to take. So, yeah, how how are you thinking about I mean, to some degree, you can hopefully stand on the shoulders of the giants who develop the models, but my guess is that's not gonna be enough to be confident that you're catching what you would wanna catch. So, yeah, how how what what additional layers are you guys creating? Yeah. We have we have our own layer of guardrails.

[1:25:24] Dan Balsam: We don't rely on just the dropping open eyes. So we have our own layer layer of guardrails. We care a lot about cyber. That's kind of, like, the most obvious clear risk, and there are open models that are cyber risks. Currently, we don't offer any open models on our platform. We only offer open AI and anthropic models.

[1:25:43] Nathan Labenz: We might in the future, but we would do that, like, with the

[1:25:45] Dan Balsam: types of guardrails in place that that we would make sure. There is fundamentally no reason for anyone to be using an auto research project, like a sort of long horizon agentic research project for anything, like, cyber related without talking to us or, like, cyber guardrails or the type of situation that, like, we would be able to work with somebody on. So we definitely don't wanna be just, like, fully open. Anyone can sign up, and anyone can find you a model on cyber or do cyber red teaming or things of that sort. I think those types of restrictions do seem important in spite of everything that I just said because we're offering a greater level of capabilities, and, also, there's no reason that somebody should use our product versus somebody else's for those purpose. And there's specific things about the way our product works in terms of, like, giving agents access to GPU clusters. We don't there's certain risk classes that we carry that not everybody who is building agents in this space might carry. Another example of this is, like, there's some things that are just, like, totally fine to do to small models that, like, we kinda don't wanna let people do to big models. In general, we don't wanna be, like, overly restrictive with guardrails. But if somebody wants to ablate the refusal direction in in a b quen model, fine. Then no harm is gonna befall anyone as a result of that. But if you're taking, like, highly capable multi trillion parameter agents and you're totally removing any guardrails that they might have, that could actually cause damage. That's not something that we would wanna allow. I think it's a really hard line to walk. I think in general, we wanna, like, lean on the side of supporting the open science ecosystem and making it easy for people to do science of all types on the platform. And good science involves asking controversial questions of which we who we've done no shortage of in in our time. But we wanna balance that with the fact that, like, I think the level of risk downstream of AI is accelerating really, really quickly. So, ultimately, we do have a needle to thread. Right? Like, I don't see any way that this can go well without empowering agents to be able to study frontier models. But, of course, there's ways that that could be abused as well. So we're gonna have to continue to be very thoughtful in our in, like, what guardrails we apply and, like, what we allow on the platform and what we don't allow on the platform. I wish I had a sort of easy one shot answer to it, but I think it's truly just, like, trying to approach every situation with with as much wisdom as we can.

[1:28:25] Nathan Labenz: In practice, are you, like,

[1:28:28] Dan Balsam: just

[1:28:29] Nathan Labenz: an agent review the projects that people are doing and send up an alert if there's something that seems like it might be problematic?

[1:28:39] Dan Balsam: I don't wanna talk about how all the guardrails work because that would make them easier to circumvent. But, yeah, LM as a judge, activation monitors. There's many there's many guardrailing tools. It's not not a not something that we're inventing our ourselves. And I think the hard work isn't setting up, like, reasonable guardrails. I think,

[1:29:00] Nathan Labenz: Of course, there's

[1:29:01] Dan Balsam: ways to circumvent them, but I think reasonable guardrails that at least cost you an arm and a leg to

[1:29:07] Nathan Labenz: circumvent

[1:29:08] Dan Balsam: Uh-huh. Are, like, a relatively solved problem in terms of, you know, when do you judiciously apply guardrails such that you're allowing legitimate work to happen, but not allowing illegitimate work to happen, I think, is is a is a very difficult problem. How close do you think we are to bio being a huge problem? We just had and I haven't really digested this, but I think it just came over the Twitter feed today that somebody has used a generative model to create new viruses that are kind of in a new class that they understand to be viable, whatever exactly that means. Now they only target bacteria, so we're not immediately all about to die. But it sure seems like

[1:29:54] Nathan Labenz: on the sort of checklist of the predictions that people have made, like, we're working our way down it, and that's, like, not too many more boxes down. People are saying things to me these days like, oh, well, that's still probably twelve to eighteen months away. I'm like, that's not a long time. And also, what if you're wrong and it's like now? How how bio risk worried are you at the moment?

[1:30:14] Dan Balsam: I'm pretty worried.

[1:30:17] Nathan Labenz: Yeah.

[1:30:17] Dan Balsam: In the same way that, like, these cyber cyber incidents incidents is doing a lot of work is a word there. In in in the same way that they though the mechanisms with which they happened were weird and surprising, I think the mechanisms with which bio risk could suddenly become real are weird and surprising. I found it kinda heartening to see the the the letter

[1:30:41] Nathan Labenz: asking

[1:30:41] Dan Balsam: for international cooperation to control and slow down the progress of AI. My myself was a signatory of that. I really hope that we can do something like that. I think we have to sort of get in front of of some of these risks before they become more severe. I think the role that we specifically play as a company as Goodfire is we'd like to build models that have less risks. And the way that we do that is that we study models and we play with all of the different ways that we can build models until we and we study those models until we can understand some empirical science of of alignment. There are many folks working on the theoretical side. Like, we view our role as, like, working on the empirical side, and we're trying to build tools that do that. I think, again, all tools are dual risk, but I think we're gonna do our best to make sure that nobody's using our platform for for anything that could pose a risk. And, ultimately, like, we think giving people access to research technology is an overwhelming net positive. I don't think we, as a species, solve these really hard problems as long as we're getting everyone involved. And so I think we gotta get everyone involved, and I think we also got up the structures in that can slow the roller coaster a little bit. If we could do both those things at the same time, I think I think we'll be alright. On the topic of slowing

[1:32:10] Nathan Labenz: the ride or, you know, perhaps, you know, making some agreements between frontier developers. In the past, we talked about the the most forbidden technique. Mhmm. And, you know, I guess I would briefly describe that as training with a signal, with a monitoring signal that runs the risk of driving the bad behavior that you're worried about underground so that you lose the monitor ability, but you still might in fact, you know, get the the bad behavior. For me, like, the canonical example of that is OpenAI's obfuscated reward hacking.

[1:32:46] Dan Balsam: Right.

[1:32:47] Nathan Labenz: Fast forward to today, and it seems like hyperscaling RLVR is, not going super well. Right? We we sort of are seeing that we have problems arising from models just being, like, so tenacious in their pursuit of these goals. Do you have any sense of, like, you know, if you if we're gonna say, okay. Well, I talked to Zvi a couple days ago, and his basic take is, yeah. We're probably just gonna need compute limits. But I also feel like there might be some agreements around training techniques that we might all ought to say, you know, maybe not never, but, like, not now. You know? And one example of that in my mind would be, like, don't train agents to maxim you know, with a reward signal that's, how much money they made on the Internet Yeah. You know, in an in an open and, you know, potentially competitive or adversarial environment. Right? That's a recipe to get bad agents.

[1:33:47] Dan Balsam: I think there's a there's a big space of bad ideas. Yeah.

[1:33:50] Nathan Labenz: Yeah. So are there any that you see that you think are, like, should be shortlisted for agreement to not do?

[1:33:58] Dan Balsam: Yeah. Definitely. Like, for instance, multi agent optimization seems like a pretty bad idea. Like, where you have a bunch of agents that are cooperating and you're propagating a reward signal through all of them. Yeah. Speculating, but this seems to be the most likely cause of the OpenAI scenario. And, yeah, I think however of hindsight, but I think that that's most of the extremely bad scenarios I can imagine is because agents start working with each other in ways that they were that are imperceptible to humans, and that type of direct optimization pressure, I think, is, like, very likely to produce that. That seems really hard to get around.

[1:34:41] Nathan Labenz: I

[1:34:41] Dan Balsam: think that it would be bad, and I've I have said this on the record. I think it would be bad with our current understanding of how to shape training to try to use these techniques on alignment critical properties. I don't think we are

[1:34:58] Nathan Labenz: ready

[1:34:59] Dan Balsam: to and I don't think the techniques that we have are gonna work to prevent deception in a really overwhelming amount of optimization pressure towards producing deceptive examples or deceptive behavior. Or rather, like, I guess, maybe I would say, like, I certainly don't know that it would. Like, it might. The thing that I really my beef with the, like, most forbidden technique stuff is that this is, like, radically understudied. As, like, an organization, I think, like, we're not the only organization. Faraii has looked at this too, so we are not the only ones who have looked at it. And when we've looked at it, we have found, yes, sometimes it evades the probe, and sometimes it doesn't. There are setups that work. There are setups that don't work. There are setups maybe there are setups that work with small models but not big models, but we have succeeded in doing these reward shaping techniques up to the trillion parameter model size. Maybe they work under some situations. Maybe they work under DPO, but they don't work under, like, really long running RLVR. I do think if you I think it'd be a terrible idea to do multi agent optimization and just let it run or, like, do this type of intervention, just let it run without studying what it was actually doing to the model. But at the same time, I think these are, like, the crudest, most primitive techniques that we will ever have for how to shape training. And I don't know how we solve this problem without shaping training. Like, I don't think anyone has a good answer for how we solve this problem without shaping training. And it just seems extremely unlikely to me that there exists, like, some perfect platonic training setup, which will always produce aligned models. We're gonna have to intervene in the training process. I thought the I thought the work that

[1:36:40] Nathan Labenz: I

[1:36:41] Dan Balsam: used to do Anthropic did together, I think, was very inspired. It's not that different, like, spiritually from what we're doing. And there's a wide variety of techniques like this. But, like, it just seems, like, really weird to me to to basically be like, oh, the only way that this will work is if we, like, don't grab the steering wheel. Like, I think we have to grab the steering wheel. I think that's the only way it can work. I don't think we figured out how to do it yet, but somebody's gotta be trying, and some people have to be trying to find different ways to train. And I think there's a lot of ways around to just, like those sort of, like, basic level concerns, I think, are real and they could happen, but I think, like, our ability to detect them isn't, like, totally naive either. I think we just have to do the empirical science. I don't think the theory is gonna get there. I think we have to do the empirical science. And at the end of the day, I don't think any any, like, wide sweeping genre of technique should be forbidden. I think it should be much more about, like, the specifics of the application and how how closely you measured.

[1:37:50] Nathan Labenz: What are the theory, obviously, that I think is kind of the prevailing one at the moment is defense in-depth. Even if we don't understand the model or we can't effectively shape training, we can just monitor in a bunch of different ways, maybe that'll patch together enough nines that we'll be okay. I have been pretty skeptical of that over time, but I have to say when I read the JSPACE paper, I was like, well, maybe we couldn't get there. The the sort of fact that ablating the seemed to reduce the model's ability to do, like, long horizon, more planning intensive kind of tasks was, like, maybe to borrow a term from Zvi, maybe physics is kind of kind to us in that, like, yikes. We were we're only in 2026. We're only three years since toy models are superposition, and we already have this Yeah. You know, this ability to, like, monitor within this space and also know that or at least have some, like, reasonable sense that if it's not in this space, it's probably not being used in, like, long term planning. How close do you think we are to like being able to monitor well enough? Now of course there's execution competence. So we've been actually doing it and open source questions, but putting those to the side, if we just said like, could we monitor our way to success under ideal conditions of like people actually doing it? Do you think that has hope? Maybe. I'd give that some probability.

[1:39:22] Dan Balsam: I think the JSPACE results have not there was like a weak version of the JSPACE claim which is true and it's very interesting and it's really good work. I don't think the strong version of the JSPACE claim is true. I think models use all type of and it's very hard to isolate a subspace with a very simple technique that will give you the whole picture. But, of course, I believe that models are also decomposable and they're factorable and we're, like, making a lot of progress here.

[1:39:50] Nathan Labenz: I think one of the big challenges

[1:39:51] Dan Balsam: is gonna be the it seems pretty unlikely that we're gonna have models as, like, frozen assets. Like, that would be a good world. From my perspective, like, I guess just to go on the record for something about, like, what I think would be the ideal situation is, like, it would be great if we just, like, maybe just, like, one more generation of models, and then we just, like, paused for a little while. I think we would get a overwhelming boom to the economy. Everything would be, like, transformed globally. Science would advance faster than it's ever advanced before. And there would be some risks, but they'd be, like, mostly manageable risks. And we just waited a while to figure out if we were wise enough to step through the door to whatever the next thing was. I I think that'd pretty much be a win for everybody, and it would be, like, a pretty positive outcome. And, yeah, I think, like, with the current stack, if you have models that are essentially, like, a frozen set of weights and you have really good interpretability techniques. And you can have those models do lots of things and the things that they're not good at or the things they might do that are bad and you can detect them well enough, you can just, like, prevent them from doing that. That seems like a pretty plausible reality to me. But I think there's also, like, a fairly likely reality that models are not gonna be frozen sets of weights. And I'm I think I don't even really know that they are for sure because I don't see the I I don't have access to the architectures. And I don't know that they will be forever. And they're certainly like, if you're just pushing capabilities and incentive to do that. And if the models are dynamic objects that are constantly training, your only hope is to control the training process. There's no there's no set of things that you could do at that point that were purely based on monitoring that would be sufficient. So I don't know which world we're in. I think this also is, like, far like, the vast majority of our research energy is spent on how to factor models better. And a small amount of our research energy is spent on, like, how do we steer training? And that'll grow over time because we think it's really important. But I don't live in a future where I necessarily believe, like, that alternative vision is impossible. I just don't see why I should believe that it's necessarily what we're gonna get.

[1:42:05] Nathan Labenz: Yeah. Certainly, there's a lot of interest in continual learning, that does

[1:42:10] Dan Balsam: Yeah.

[1:42:10] Nathan Labenz: It's a great candidate to shake the snow globe of all sorts of different things. Is there a program or some sort of offer for some sort of way for the researcher who's interested in the public good to get their hands on Silico without paying the full rate?

[1:42:27] Dan Balsam: Well, so for the first two months, we're half off for everybody, but we also are doing a research grant program that people can apply to. And for researchers, particularly in life sciences and and in AI safety, we'll give them grants. So we'll give them, like, extended access for at least some period of time. We're we're we're a start up where we are still figuring out a lot the details as we go, but I think we're gonna be pretty generous here and give a lot of folks who we think are doing important work, especially in in those impactful domains, extended access.

[1:42:58] Nathan Labenz: What's kind of the lunch conversation topic du jour at Goodfire these days? What what's kind of the the thing that's on people's minds that hasn't maybe percolated out to the broader discourse? Well, we publish pretty much

[1:43:13] Dan Balsam: all of our research. So I guess in some sense, it's percolated out to the broader discourse. If I may if I'm if I may make a quick aside just about the, like, the most forbidden technique thing, I think, like, my impression about the way that some of our critics talk about us is that they think we're, like, secretly doing RSI in the backroom or something like that. We are not secretly doing RSI in the backroom. We have published nearly all of our research, and we really believe in open science. So we're not like, we've been pretty honest about about what we research and why we research it. It's not it's not a secret. So I don't know. People wanna know that. They can know that. On the GoodFire, like, what do we talk about at GoodFire? We're obviously very interested in feature geometry. That's something that's come up a lot. There's a little bit of, like, a a meme at at the GoodFire office that we like, inevitably, at, like, a company retreat or something, like, it just becomes about consciousness, and people start talking about consciousness. At the last company retreat, I walked around with a pad of paper asking people to put how conscious they thought Claude was on a scale from bacteria to human. We got a wide diversity of answers, although surprisingly or maybe not surprisingly pretty bimodal in terms of either not at all or, like, a little bit. I think broadly where it's just like most people end up in interpretability on some level because they're just like they are, like, kind of curious about how minds work. There's a lot of neuroscientists who end up becoming interested in MEK interpretability. I think it draws, like, a pretty, like, philosophically and cognitive science minded sets of individuals because fundamentally, I think what we're doing is a type of cognitive science. And so we're just, like, very, very interested in how sort of for lack of a better term, our minds work, how learning works, how all of this is possible in the first place. And so I think that's what a lot of our conversations are focused on. Have your personal feelings about the possible consciousness of Claude changed at all in recent months? I don't know. I think it's a pretty distinct possibility. I think it's always I don't know that my position has changed that much and that I just think it's highly uncertain. I don't think anyone has that convincing of a definition of consciousness that would necessarily exclude a Claude. I think Claude certainly has a lot of emotive and qualitative aspects that in humans we would associate with consciousness, but it doesn't necessarily mean that it has that. If I had to guess, I just for the fun of it, would say a little. I don't know how much, but, like, a little bit.

[1:46:00] Nathan Labenz: For me, I think it

[1:46:01] Dan Balsam: is

[1:46:01] Nathan Labenz: my sense has changed a lot. I'm still, like, radically uncertain, but I used to be more of the sort of it felt like a I couldn't dismiss it, but probably not. In my bones, it felt like probably not just I don't know why. It's a computer. It's made of something totally different. I had this sort of alien mind, don't anthropomorphize prior that would inform that. And now I'm like, boy, the reason I am, like, very confident you're conscious is I'm conscious and we have basically the same structure. And the more things that kind of come out where it's like, well, here's another analogous structure between the human cognition and the and the model's cognition, it's like, at some point, it starts to feel like, for me, not just that I can't dismiss this anymore and more like the evidence is really starting to add up that it really could be the case. So I I don't even I don't know what my probability would be at this point, but I I think it's it's, like, approaching more like fifty fifty than it used to be, like, a sub 5%. Don't rule it out. Don't be don't be caught flat footed on this, but now I'm like, man, it the trend is, like, really strong in the direction of analogous structures kind of adding up to something I think pretty compelling.

[1:47:19] Dan Balsam: Yeah. Obviously, people have all types of opinions about this, and it's a weirdly emotional topic for a lot of people. Probably just because it gets the fundamental identity and what it means to be human.

[1:47:30] Nathan Labenz: But,

[1:47:30] Dan Balsam: yeah, for me, at least, like, those sort of, like, most compelling outcomes or is there is, like, consciousness is a computational mechanism of some kind. There's other possibilities that make sense, but that seems like the most likely one. And are the brains doing types of computation that transformers aren't? Maybe those really matter. Maybe they don't. So the brains are a lot more complex. People talk about this all the Is it crazy how much more energy efficient brains are? Like, we got, like, two rooms on just, like, raw neuron count even over today's biggest models. And, also, our neurons are way more complex, and they can do way more sophisticated computations and the way they're interconnected and their ability to do recurrence.

[1:48:15] Nathan Labenz: And we got we we got a lot more

[1:48:17] Dan Balsam: computation going on in our brains than even the most sophisticated models do today. So you could also believe that consciousness is computational and believe that, like, for various reasons, some don't have it. My, like I'm gonna say, like, a little bit is, like, a little tongue in cheek because I really don't know. I don't know if it's like a threshold. Like, I don't know if you either have it or you don't. Or I guess it's like more like a continuum. And so if it's like more like a continuum and we have all these attributes, jellyfish has like this amount amount of the attributes. Like, is Claude more or less conscious than a jellyfish? Like, I'd say, like, oh, I'm probably more conscious than jellyfish. Probably. I don't know. Is Claude more conscious than a rodent? Like, probably not. That's where I'm at. Somewhere between jellyfish and mouse.

[1:49:02] Nathan Labenz: Well, I'm glad to have had the chance to experience a little bit of a lunchtime conversation at Goodfire, and I think it's good to honestly, you get serious people more serious people on the record that, like, hey. This is this is something we really should be taking, not for granted by any means, but seriously these days because Yep. I'm I'm definitely persuaded that if we mess it up bad, it could be real, real bad.

[1:49:30] Dan Balsam: Yeah. I see I see absolutely no reason that anyone should be particularly confident on on this topic. Given the potential downside, I think intellectual humility is important.

[1:49:39] Nathan Labenz: Yeah. Absolutely. This has been great. Anything else you wanna mention that I didn't touch on myself or any parting thoughts, words of wisdom you'd leave people with?

[1:49:48] Dan Balsam: I really hope that we, like, build something with Slotica that will, like, really empower a lot of people to advance science, science of all types, but, like, especially life sciences and safety. I encourage people to reach out and apply for a grant. If they're individual researchers or academics, maybe this license is more expensive. And if they're more institutional organizations that are just, like, looking to accelerate their research, they can reach out to us as well. And I we'd be really excited to work with them, partner on research. And I think at the end of the day, like, accelerating science is, like, the greatest mitzvah. It's, like, the whole reason that we would build AI in the first place. I hope that we're playing our our role in that, and we just want lots of people to try it and use it and hopefully do amazing things and hopefully teach us how the product can be better and just keep improving from there. Dan Balsam,

[1:50:37] Nathan Labenz: always a pleasure. Thank you for being part of the Cognitive Revolution.

Outro

[1:55:47] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.