How Agents Decide: Goodfire's Eric Bigelow on Critical Tokens, Phase Shifts, & In-Context Learning

Watch Episode Here

Listen to Episode Here

Show Notes

This episode started with a question Nathan couldn't shake after his conversation with Bronson Schoen of Apollo Research. Even someone who has read more chain of thought than almost anyone often can't predict what a model will finally decide. So what do we know, at the level of mechanistic interpretability, about how that decision actually gets made? Nathan asked AIs for a literature review, and they kept coming back to one name as the throughline: Eric Bigelow, now a Member of Technical Staff at Goodfire, with a Harvard Psychology PhD and the dissertation Towards a Cognitive Science of Large Language Models. As far as Nathan knows, that makes Eric the first guest in roughly 400 episodes who was suggested by an AI.

Eric's starting point is Forking Paths in Neural Text Generation. He and his co-authors took GPT-3.5 chain-of-thought answers, resampled about 30 rollouts at every single token, and tracked the distribution of final answers over time. The result echoes the phase transitions of grokking. The answer distribution often holds steady and then suddenly collapses. Sometimes that happens where you would expect, at the first mention of an answer. Sometimes it happens on an arbitrary token, like the open parenthesis in "kilowatt hours (kWh)". His short answer to Nathan's question is that the decision "really happens during sampling." In his account, reasoning is the model in-context learning from its own sampled tokens. A coin flip changes the context, and everything after it stays consistent with that flip. He uses the broad definition of in-context learning from Lampinen et al.'s "broader spectrum of in-context learning": anything the model does to adapt that isn't in-weights learning. His story work (Stories in Space, also on Goodfire's blog as Meandering on Manifolds) was inspired by Kurt Vonnegut's "shape of stories." It shows that averages over many stories look smooth, while an individual story's belief trajectory can turn sharply after a single sentence.

The conversation then turns to how models have changed since the GPT-3.5 era. Eric regrets that the frontier APIs have hidden their sampling parameters. He describes SFT and RLHF as sharpening the output distribution, and RL as pushing it somewhere new. A heavily RL'd Qwen3-30B-A3B writes about a clockmaker in about two-thirds of its stories. GPT-6 Astra, by contrast, writes varied stories, probably because it enumerates options in its reasoning and then picks one. That leads to one of the episode's central claims: "reasoning is almost like a misnomer" for what reasoning models do. Models like DeepSeek-R1, which can say "wait" 50+ times in one chain, produce what Eric calls "a soup of tokens" and then "skim something out of that soup." Forking-paths curves on reasoning models are smoother, but they still have forks. Eric's own Reasoning Theater work on performative chain of thought comes up as Nathan asks whether phase changes happen before, during or after the relevant reasoning. Eric's answer is all three. A common pattern is stable uncertainty right up until "the answer is…".

On which models to study, Eric's rule of thumb is that interesting zero-shot behavior starts around 7–8B parameters, and he treats frontier open-weights models as reasonable stand-ins for proprietary ones. At the very top, Kimi K3 has no American equivalent. Goodfire has found reward hacking in 50–96% of rollouts across Kimi K3 and other leading open models, a behavior that only shows up at that scale. Eric thinks a ban on Chinese models would slow frontier-scale interpretability. He thinks a ban on open-weights models generally would do far more damage. At smaller scales he could switch from Qwen to Gemma without losing much.

The cognitive-science thread runs through the whole conversation. Eric describes two healthy ways to anthropomorphize. One is loose, for your own understanding, like saying your car is "trying to tell you something." The other treats words like belief, decision and intention as serious research targets, and he invites critics in psychology, linguistics and philosophy to help rather than scold. He retires the stochastic parrot as a shallow, Chinese-room-style argument, while insisting that stochasticity itself is essential. He cites experiments in which people rationalize choices they never made (the classic choice-blindness studies) as a sign that humans have forking paths of their own. His working account of belief is Bayesian inference over latent concepts, developed in Belief Dynamics and The Shape of Beliefs. Unlike brains, LLMs let you look inside and intervene. The Road Not Taken line of work suggests models don't represent full future rollouts. Eric's guess is that they represent a few steps ahead plus a rough "shape" of where they might go, the way a mathematician can see a proof's outline before deriving it.

The last stretch looks ahead. Open looped models like Ouro still show uncertainty that shifts at particular tokens, so latent reasoning doesn't remove the coin flip. Eric wants interfaces like Janus's Loom that show where a response could have branched. On why chain of thought is drifting into strange dialects, including how Astra talks to its subagents, he points to outcome-only RL, language evolution and Goodhart's law. His confidence in chain-of-thought monitoring has fallen, partly because of the "stolen chain of thought" paper. His prescription is simple: far more investment in understanding models. He ran the Forking Fast experiments entirely in Goodfire's Silico agent in a week. His advice for working with research agents is to give them rich context, explain your reasoning, and keep your own scientific taste in charge. He closes with the topics around the Goodfire lunch table: automated interpretability, the company's shift toward alignment since the Hugging Face incident, and what "reward hacking" even means.

Topics covered

• An AI-recommended guest: the literature review that named Eric as the throughline
• Forking Paths: resampling at every token, sudden collapses in the answer distribution, and the "(kWh)" fork
• Reasoning as in-context learning from your own sampled tokens; the broad view of in-context learning
• Stories have shapes: belief trajectories while reading, smooth on average and jumpy in individual stories
• Continual learning vs. context engineering, and whether interpretability tools will carry over to personalized models
• Hidden sampling parameters, output diversity, and what SFT, RLHF and RL each do to the distribution
• Reasoning models as linearized tree search, or a "soup of tokens"
• Choosing models to study: the 7–8B sweet spot, Kimi K3, reward hacking, and possible bans on Chinese or open-weights models
• Anthropomorphism done carefully; retiring the stochastic parrot
• What a belief is in an LLM: Bayesian latent concepts, and what models represent about their own future
• When phase changes happen relative to the reasoning, and interfaces that show branch points
• Looped models, latent chain of thought, and why sampling still matters
• Why chain of thought and agent-to-agent language are drifting from English
• RL-scale vs. personal-scale representation change
• The plan: more interpretability research and far more investment
• Doing research with Silico: context, hand-holding and scientific taste
• Around the Goodfire lunch table: automated interpretability, alignment, reward hacking, latent CoT

Resources

Eric's research

• Forking Paths in Neural Text Generation (Bigelow, Holtzman, Tanaka, Ullman; ICLR 2025)
• In-Context Learning Dynamics with Random Binary Sequences (ICLR 2024)
• Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics (Zur, Geiger, Lubana, Bigelow)
• Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
• The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds (Sarfati, Bigelow et al.)
• Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space · Goodfire post
• Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation · arXiv
• Reasoning Theater: Probing for Performative Chain-of-Thought · arXiv
• Eric's site · X · Google Scholar

Goodfire

• Goodfire · Silico
• Models know when they're reward hacking — and we can catch them at scale
• Steering Along Manifolds to Control Neural Networks

Referenced in the conversation

• The broader spectrum of in-context learning (Lampinen, Chan, Singh, Shanahan)
• Loom: interface to the multiverse (Janus)
• Ouro: Looped Language Models · Scaling Latent Reasoning via Looped Language Models
• Othello-GPT: Emergent World Representations · Othello-Mamba
• Progress measures for grokking via mechanistic interpretability (Nanda et al.)
• DeepSeek-R1
• Stealing Reasoning Traces from Proprietary LLM APIs · stolen-thoughts.com
• GPT-6 Astra system card
• Kimi K3 · Qwen3-30B-A3B · Gemma · Llama 3.1 8B
• Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident (Hugging Face)
• On the Dangers of Stochastic Parrots · The Chinese Room Argument
• Bayesian Models of Cognition (Griffiths, Chater, Tenenbaum)
• Failure to Detect Mismatches Between Intention and Outcome in a Simple Decision Task (choice blindness; Johansson et al., Science 2005)
• Tomer Ullman · NTT Research Physics of AI Group
• Kurt Vonnegut on the Shape of Stories (link?)
• Neel Nanda, "phase transitions are everywhere" (link?)

Related Cognitive Revolution episodes

• RL's a Hell of a Drug: Bronson Schoen of Apollo Research
• Thinking in Silico: Goodfire CTO Dan Balsam

Quotes worth pulling

"I think this decision really happens during sampling" — Eric Bigelow

"I really think reasoning is almost like a misnomer for what reasoning models are doing." — Eric Bigelow

"it's almost like a soup of tokens they create, and then they go back and skim something out of that soup." — Eric Bigelow

"I think in-context learning and reasoning are unfortunately vastly neglected in the world of interpretability, compared to how defining they are for how we think about a lot of the high-level things that LLMs do." — Eric Bigelow

"But we don't have that excuse with LLMs. It's very easy to peek inside the "brains" and do interventions" — Eric Bigelow

"I think the stochastic-parrot metaphor should probably be put to rest." — Eric Bigelow

"if we're only giving them reward or punishment based on their final outcome, then they might as well speak to their sub-agents in weird, non-human languages." — Eric Bigelow

"for the amount of investment happening in making models better, versus the amount of investment in understanding them, there's just no comparison." — Eric Bigelow

"Research taste and scientific taste is everything. It's worth its weight in gold." — Eric Bigelow

"I think using Silico is kind of like working with a very eager, first- or second-year PhD student who has a lot of skills and is hardworking, but also doesn't always know what it's doing in the bigger picture" — Eric Bigelow

Sponsors:

Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

CHAPTERS:

(00:00) About the Episode

(03:53) Forking paths in reasoning (Part 1)

(15:53) Sponsors: Parallel | Claude

(18:47) Forking paths in reasoning (Part 2)

(19:40) Dynamics of in-context learning (Part 1)

(28:16) Sponsor: Tasklet

(29:53) Dynamics of in-context learning (Part 2)

(29:56) Continual learning and interpretability

(38:55) Shifting distributions through RL

(46:41) Selecting models for research

(57:11) Cognitive science and LLMs

(01:07:35) Modeling beliefs in AI

(01:16:22) Phase shifts and uncertainty

(01:31:49) Uninterpretable chain of thought

(01:45:57) Automating research with Silico

(01:50:57) Safety and alignment frontiers

(01:57:07) Episode Outro

(02:00:02) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution!
Today, my guest is Eric Bigelow, Member of Technical Staff at Unicorn Mechanistic Interpretability startup Goodfire. The story behind this episode is unique. After recently speaking with Bronson Schoen of Apollo Research, who explained that, even with access to models' internal Chain of Thought, it's still often extremely difficult to determine how a model will decide to act, I asked Claude to survey the literature to see what the field as a whole understands about how "critical tokens" are chosen. One name kept coming up in that thread, and it was Eric. Eric recently completed a PhD in the Harvard Psychology department, with a dissertation titled "Towards a Cognitive Science of Large Language Models," and while some of his academic colleagues initially questioned his pivot to focus on LLMs, the fact that even elite academic institutions are now willing to engage with LLMs as a sort of mind strikes me as very notable. We start with a survey of Eric's work over the last couple years, beginning with his 2024 paper on "Forking Paths in Neural Text Generation", where he conducted a massive resampling experiment at every token of a model's chain of thought, to identify critical tokens where uncertainty around the final answer suddenly collapses. As so often with LLMs, some of the findings were intuitive, while others were quite surprising and difficult to interpret. Interestingly, Eric says that a system's decisions are actually taken not so much by the model itself, but by the process of sampling from the model's final token distribution, which means that while the stochastic parrot meme should definitely be retired, stochasticity still plays a meaningful role in model behavior. Eric frames much of what we are seeking to understand about LLMs in terms of "in-context learning", emphasizing that few-shot prompts are just one clear example of how much runtime context can shape model behavior. His bet is that this flexibility means that improved harnesses and memory systems will outcompete per-user fine-tuning, and thus that our focus should remain on major foundation models. Along the way, we discuss: * why 7-8 billion parameters currently seems to be the interpretability sweet spot, * why Kimi K3 was a step function for studying frontier-level coding and reward hacking, * and why, for now, interpretability outside the frontier model companies does meaningfully depend on Chinese open-weights models. We also get Eric's thoughts on why Chain of Thought is getting so weird, including his perspective on the chain of thought dialect and the strange language Astra uses to communicate with subagents, plus some pro tips for using Goodfire's Silico research agent, and how Eric hopes future LLM product interfaces will begin to re-surface model uncertainty to users. The bottom line, from this conversation and others I've had recently on the future of AI monitoring and control, is that chain of thought is increasingly unreliable, and that because these issues seem to correlate not just with RL training but with the power of the raw pre-trained model itself, this trend might be quite difficult to reverse. That in turn means that an awful lot, including our ability to understand why agents take the actions they do, may rest on interpretability. Eric is a pioneer of such research, but as you'll hear, tons of work remains to be done. With that, I hope you enjoy this overview of what we know – and how much we have left to discover – about how AIs decide what to do, with Eric Bigelow from Goodfire.

Main Episode

[03:53] Nathan Labenz: Eric Bigelow, member of technical staff at Goodfire. Welcome to the Cognitive Revolution.

[03:58] Eric Bigelow: Thank you very much, Nathan. Excited to be here.

[04:01] Nathan Labenz: Yeah. I'm excited for this conversation too. I think this might be the first time in the history, and there's been, like, 400 episodes of the podcast at this point, that I went to AIs looking for an answer for a very specific question and came back with a name, and that name was yours. So the kind of upstream conversation I had of this one was with Bronson Shane at Apollo, who has the dubious distinction perhaps of having read maybe more chain of thought reasoning than, like, any other living human, least at any living human who can talk about it all publicly. And one of the big takeaways I had from that conversation was he said, you look at these chains of thought, and they're... The the models are thrashing around doing a sort of linearized tree search of possibility. And then at some point, they make a decision. And even having read all the chain of thought that came before that, it's often, like, very not obvious why they made the decision that they made, and you really couldn't have predicted it. And in a way, that's, like, kind of human like. Right? It's how do I make my decision? I'm not so sure I have great insight into that all the time either. But this prompted the question, okay. At some point, after all this stuff, all this reasoning, all this... And trying to reason about what the greater wants and all that stuff, Eventually, we come to a passage, it's like... And so the answer

[05:22] Eric Bigelow: is,

[05:24] Nathan Labenz: how does that next token get chosen? That's what I went to the AIs to ask. I asked for a literature review, and Eric Bigelow came back as the through line through that research. And I was like, I know that company. I should get to know that guy. So what I wanna do, I think you may know as much or more than any other living human about how these decisions are made. And I'm afraid that's maybe not still all that much, but what I hope to get out of this conversation is little intellectual history of your work and, like, the up to date understanding of how the hell do these things make decisions? What do we know about that? And what is our plan to figure out how to understand it better? Because, obviously, they're making more and more consequential decisions all the time. How does that sound?

[06:08] Eric Bigelow: I'm very honored to be the first guest chosen by the AIs. I always... I feel like they when I talk to Claude, it seems to really get excited about my research, but I'm sure everybody feels that way. I think I think actually talking... Having some conversations around around this podcast and some of the things you're asking about makes me think that I actually should devote more effort to this exact question of how decisions are made. But I'll give a quicker review of project that I think is really one of the most informative projects in how I think about how LLMs work at a basic level. And that's my project Forking Paths in Neural Text Generation. And for that, basically... So this was two years ago with... At the time, it was one of the GPT 3.5 models. And what we do is we take reasoning chains. We give it a chain of... We give it a question and a chain of thought prompt and have it reason through, you know, step by step. And then every single token in its reasoning, we resample all these completions. So we resample, like, 30 different rollouts at every single token. And then from every one of those rollouts, we see what final answer it landed on. And you can aggregate these into various distributions and then do analysis on these distributions. But I think the most striking part of this is if you arrange it like a multivariate time series, you see these really interesting dynamics that are happening. And I kind of take inspiration from this from some of the learning dynamics work that was coming out around that time, like, around grokking and things like this where there are, like, these really sharp phase transitions in in training, and the model just seems to suddenly learn something. And it turns out that you actually have these during reasoning and in context learning as well, where over the course of reasoning, there will be certain points in reasoning that the model was initially sort of at a 50% chance of this answer and 30% chance of this answer. And then there's some point that it hits where suddenly the the distribution changes really dramatically. And often, the distribution just kind of collapses onto a final answer as the model sort of, you might say, decides what it's going to say next. Although I think the... I I use the word decides here with maybe scare quotes around it because I think there's some nuance to what that means. But one of my big takeaways from this... And I think this also just kind of falls out of thinking about the architecture, but it's nice to have results that really support this too, is that models, even to that some extent when they plan out their responses, what they're actually going to say at every single moment is really determined by, like, the flip of a coin, which is what token will it generate next. And there is sort of an interface that was really standard in... With OpenAI at least at the time, which I think is unfortunately no longer present, and I haven't really seen standard interfaces, which is where you could see the log probabilities of all the tokens in a text output you got, and you could kind of hover over different tokens and see what are the alternate tokens that could have been generated instead. And if you do this with a lot of text generation, you might realize that you get this one text answer when you ask some question, but really, there actually could have been a lot of different answers. And you might ask what would have happened if this word was different from this word. And what we found is that there are certain keywords where sometimes the words... If you were to sort of, like, look at it like a judge would, you might guess this, like, the first time that a final answer is mentioned in the reasoning chain. But then there's other times where this distribution changes really dramatically on some sort of arbitrary word, even like a open parenthesis for, like... You know, there's one example we have in the paper where it's like kilowatt hours, open parenthesis, k w h. And, you know, you never expect that to really change how the final answer plays out. But if you actually go to that exact token with this model and you just resample a bunch of text generations, you see that in cases where it's not an open parenthesis, where it starts to be some other just arbitrary word, it ends up being a different final answer. And so I think this this thinking about decisions in LLMs or what they're gonna say next very much being determined by sort of chance and by whatever tokens are sampled is really, I think, the foundation for how I think about a lot of things in LMs.

[10:41] Eric Bigelow: And I think it sort of shapes how I think about, like, beliefs and and models and sort of what's considered like a factor hallucination can be dependent on what tokens were generated before it. And and I think one of the properties of reasoning that really comes into play here is self consistency. So if there's a long reasoning chain that is self consistent and it sort of and it sort of generates some token earlier in the chain of... Suppose that this was the year instead of this was the year. If if it thinks that the year is 2024 and '20... Instead of 2026, actually, the answer to a different... To a question might be very different. And there's cases like this where it's like, you you might think, well, obviously, there's, a ground truth. If I'm asking who is the current head of state, there is a current year, which right now is 2026, and so the model was hallucinating. I But think there's also a lot of use cases with LLMs where we don't really know what the ground truth are. And and there's sort of this this branching space of possibilities that we actually wanna explore naturally when we're interacting with a model. And, like, for example, if I'm working on a scientific research project and I wanna sort of talk to my all m about, like, what should I do next or what what do my experiment results mean, I might really wanna kinda, like, iterate and understand what's this full, like, space, this whole tree of possibilities that it could have generated, and what would have been different if it had assumed a instead of b. And so to to go back to your question, I think the the simple answer I have is that I think it... This decision really happens during sampling is my biggest part of how I think about this is that, again, every every token that's generated them by the model to... I... One of the other big influences in my work is thinking about in context learning and how models adapt to context. And the way I think about reasoning is that models are basically in context learning from things that they have generated already. So the words that they generate, there's sort of this stochasticity that's happening. There's these coin flips that are happening as it's... You could say deciding, but it's really just a sampling from a distribution of these alternate tokens. But depending on what token is sampled, it in context learns, and there's this context dependence where then it'll generate a set of reasoning that is consistent with that. And I think... I I I think that, like, I'm pretty bullish around, like, Mechanter shedding more light, I think, on really answering this question of how decisions are made. But then I think I think so I said that this work was done a couple years ago, and I think that modern reasoning models are a little bit are a little bit different than chain of thought reasoning was then. And some recent projects I've worked on even with, like, McIntyre of reasoning models have kind of changed a little bit how I think about what these models are doing. And so I think I think there's still context dependence, but actually as you as you mentioned, reasoning models, I think, often tend to do this, like, really, like, linearized search where they, like, enumerate all these different possibilities in in context. And then at the end, they sort of just go back and pick one of the things that they said before. So I think this this might change exactly how this happens. From what I've seen, the uncertain... The the uncertainty dynamics, if you do this, like, forking path analysis on reasoning models, is often a bit smoother, but there are still sometimes these forking points where if one token is generated different from another, where if the model was really... It doesn't... You know, it it enumerates all these reasoning chains and none of them really seem that much more promising than another. Then eventually, it does sort of... There are cases where it will just say, okay. The answer therefore is b, c, and and it actually could have been different depending on what was generated. But I think there could still be more work done in in in understanding the internal processes that that lead to that that point of choice even. And and another project that I'm on has has shown that there are, at least with some some other toy domains, there are these cases of, like, neurons that seem to, like, track model confidence where you can even, like, steer and make the model more or less confident generically about whatever concept it's learning. But, yeah, I I think I think we really need more interpretability work in this vein, and I think in context, learning and reasoning are unfortunately vastly neglected in the world of of interpretability compared to how, I think, how defining they are for how we think about lots of the the high level things that LLMs do.

[15:15] Nathan Labenz: Yeah. I mean, the... When you talk to the guy who's read more chain of thought than anyone and you keep in mind that the... A huge amount of the Frontier companies plans right now rests on chain of thought monitoring as, like, the way we're gonna keep track of what the AIs are doing and if they're doing the right or wrong thing. And then you hear that... Yeah.

[15:35] Eric Bigelow: Even having read it all

[15:37] Nathan Labenz: and being, like, fluent in these, like, strange internal dialects that they use, I still can't tell you why it chose what it chose. It's, oh, yikes.

[15:44] Eric Bigelow: Where's the the red phone to call the interpretability department?

Sponsor

[15:53]Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

[17:12]Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Main Episode

[18:47] Nathan Labenz: I think there's a few different directions you started to touch on there that I would like to follow-up on. Maybe for starters, let's go on to this relationship between in context learning and other things. Right? Like, we... There's been a lot of work that's looked at, if not... This is maybe too strong, but sort of an isomorphic relationship between in context learning and fine tuning. I think that's, like, especially true at LoRa scale fine tuning where you can localize the the weight changes. Maybe it's also true even if you're doing, like, full weight fine tuning. I'm less sure about that. How should we understand that relationship? What do you think is the most important that may not be obvious?

[19:40] Eric Bigelow: So I think I think the most important thing I'd say about in context learning that may not be obvious for people, especially if you're if you're just getting started learning about it, is that I think when people say in context learning historically, it's meant this sort of traditional few shot... Give a few shots of input output examples and see how the model behavior changes. And you can do these, like, scaling curves, for example, where you, like, increase the number of shots that you give and see how model import... Performance improves as a function of this. But then I think the way I think of in context learning... And I think there's a few there's a few other people that have have echoed this. And I I... One one paper that I like around this is by Andrew Lambert and others on the... I think it's called the broader spectrum in context learning, where I think... What I think of in context learning is capturing is everything the model is doing to adapt its behavior that's not happening within weights learning. And so this is how it's changing its representations as it reads each new word of input text. And this also applies to reasoning where it... With reasoning, it's generating text and then using that to determine its following behavior. And and so I think I think this view of in context learning is sort of being really, like, ubiquitous. I I think it it gives some teeth to thinking about prompting where, like, there's this term prompt engineering that is sort of a little bit of, like, a dirty word, I think, scientific community where it feels like people just come up with, like, a bunch of hacky solutions that make the models go better. You know? Tell it you're a brilliant software engineer and it works, and it writes code better, these sort of things. Although these are maybe a bit less prevalent today because maybe we've kind of, I don't know, fine tuned some of that some of that distribution shift just into the models. But I think I think there's a way that it really defies comparison to, like, this kind of in in weights learning where really prompting is that's not gonna disappear no matter how much fine tuning you give. You're gonna have different context that you give the model, and you want it to adapt its behavior dynamically. Right? Like, think even if you... I think fine tune a model enormously, like, still want it when it's, like, reading when it's reading Harry Potter for the first time. I mean, of course, it's in the training data, but suppose there is another, you know, story it was reading for the first time. You'd want it to be updating its beliefs internally about what's going on in the story. Who are the good guys and the bad guys? Who's... Who is happy and who is sad at different times? And really, like, I I have another paper around this sort of a belief updating representation change in stories. And I and I kinda like this metaphor because I think it... It's easy to understand that, you know, as you read a story, your beliefs should change. There's there's a bunch of work around analyzing the sentiments of stories. And and one of the inspirations from that project was... There's this famous talk by Kurt Vonnegut on the shape of stories where he's describing, like, these different stories, like Cinderella starts the... Cinderella is very sad. Everything's horrible. And then she gets invited to the ball, and everything's great for a little while. And then the and then the the magic sort of falls apart, and she has to go back to her normal life, but a bit better than she was before. And then finally, at the end when when the prince finds her, you know, she has infinitely happy ending or or however he says it. And I think, like, stories have shapes, and really all context that we give models also have shapes that are dynamic and temporal. And I think this is something you just... It's it's hard to really compare it to fine tuning or to these other ways of changing representations. And I think more broadly, think that this is a piece of the puzzle, the thinking about interpretability of LLMs and not just feed forward networks is that everything that LLMs are doing is happening dynamically. They're updating their beliefs.

[23:57] Eric Bigelow: They're shifting representations inter... Internally. They can learn new representations in context. And this doesn't just happen when you give a few input output examples. This is happening all the time. This is happening over a conversation. This is happening within sentences. This is happening during the period where a model is reasoning or generating a really long input out... Or output text that's sort of changing its representations dynamically to adapt what it's doing in the moment. And... Yeah. I think I think a lot of work kind of misses these dynamic changes that are happening. I think a lot of a lot of interpretability work sort of focuses on these, like, templates where there's just sort of a couple a couple variables that are changed in the input and behavior is mentioned on, like, one token or one or or one one sort of output variable. And really what LMs are doing is, I think, much more dynamic and adaptive all the time. One other point I wanna I wanna sort of raise on this view of of in context learning is that something I was inspired by too at the time with with Grokking and this kind of work was this idea that that if you look at learning curves over a huge dataset, they'll be very, very smooth. And as you just keep training the model and scaling parameters, things will just very, very smoothly increase as you as you train it for longer and longer or errors will decrease rather. But then if you look at specific problems, specific problems can actually change very sharply. There actually can be these phase transitions. And and I think it was it was Neil Nanda who had this quote of, like, phase transitions are everywhere. If you just looked closely enough. If you broke your dataset down into, like, all the... Into just individual tasks and you looked at individual tasks, there'd be a lot of stepwise changes where the model really seemed to just suddenly learn something. And I think in context learning is like that too, where if you look at aggregate, if you look across a large dataset, things might look smooth, maybe deceptively so. But actually, cases, like an individual story, have these really nonlinear dynamics. And curves can change very sharply. Things can change after even just one one token or one sentence. And other properties can change not just, you know, suddenly learning something, but beliefs can change. Beliefs can go up and down and follow all sorts of all sorts of weird twists and turns. And so I think another another thing that I think is sometimes missing from use of in context learning is this, you know, zooming onto one example sort of problem. And something that I think is also, like... I'm I'm really curious if the if the person you had earlier who's looked at all these chain... Chains of thought, if you looked at, like, resampled outputs for the same model or kind of done some of this, like, resampling within a single chain of thought. Because I think sometimes you can get a lot from looking at patterns by just, like, looking at, like, ton of different, like, rollouts and just, like, trying to understand, like, how the model thinks in some general way. But then I think, like, for an individual prompt, there's sort of this, like, really elaborate landscape just in how the model will think about one problem. And I think sometimes you can only get this by looking at a lot of different resampled outputs for that same problem and seeing, like, where are the patterns, what kinds of things are similar or different. Like, if you if you give models... Depending on which model you you have, if you give them a very open ended prompt, generate a story, sometimes you'll notice these really funny patterns where they'll write very different story... Like, things will be superficially very different where it'll kind of start out, like, very different, but then they kind of... Like, I I I started out that Forky Paths project by just telling me a story about a boy named John. And, like, half the stories involve John having a dog, and they go on an adventure, and a whole lot of those involve them eventually finding pond or a waterfall or something. And then, like, sometimes the dog saves the day or sometimes John saves the day. But I think these are all these different patterns you see for outputs for these same input example. And so I I think I think we need to do more of that, like looking at, like, a lot of resample outputs for the same input examples to kind of understand, like, bumpy or smooth the landscape of in context learning is for these models. One big thing pushing toward

Sponsor

[28:16]Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

Main Episode

[29:56] Nathan Labenz: is continual learning, and I wonder what your experience trying to understand in context learning suggests or implies about how much additional challenge full continual learning might present. I could imagine... Like, I can think the naive read would be like, this is gonna be really hard now because we've got everybody's individual model becomes personalized over time and who's to say how different they are. But then the... Maybe the counterargument would be like, we already have that within context learning because there's a lot of context, and we basically already deal with that

[30:38] Eric Bigelow: same problem.

[30:39] Nathan Labenz: Does your experience make you feel like we're ready for continual learning in terms of our interpretability techniques or that it will still pose, like, a a step change increase in difficulty?

[30:52] Eric Bigelow: I think I think for your question of whether our current, like, interpretability techniques will face challenges with with continual learning, I think it really depends on how continual learning plays out. And it's something that I haven't, like, personally done research on or thought too much about in-depth on, like, do we need different architectures for this? And I know there's sort of a lot of people that have been debating this for a while. And as you're saying, like, having personalized models or having models that are, like, fine tuned to your data. I think if that's the case, then there's a possibility that we'll have to have interpretability. Our interpretability techniques will have to evolve as well, and we'll have to have some sort of, like like, I think I think how how well... You know, say say that you have a tool that's trained on a base model, and then that model is fine tuned for 10 different people or 10,000,000 different people. I think how... The real key question there and how well your tools will directly translate is how much representation change there is. And, like, I have some work on this, and and other folks at Goodfire have some really cool work on there being these low dimensional manifolds that describe the sort of underlying representation space that models are going through as you're as you're interacting with them and as... And a lot of the representations are sort of defined by this low low dimensional conceptual structure. And I think my intuition would be that as long as a lot of that structure remains the same, then a lot of the techniques will be pretty easy to translate. And maybe you need to have a bit of fine tuning of your probes so that they work on each person's particular fine tuned model in that world. But I think the only real challenge there is if the continual learning leads to such catastrophic forgetting that the internal representation structure of the model radically changes. But I kind of I kind of doubt that'll happen because that... I think that will probably mean that the models will break. Like, it may be that they... Is massive internal representational reorganization, but things are still structured and the model can still do lots of, like, all the old things it can do but in new ways. But it seems to me more likely that if if we're in a world where really there is this collapse of common internal structure, then the models are probably probably breaking in that case unless in this world of continual learning, we're using an ungodly amount of compute for every one of these 10,000,000 people to to train them on, you know, do do a lot of meta learning so that they maintain their performance on all these tasks and learn entirely new representational structures. So, yeah, I think I think in that world, it's kind of like having, like, a bunch of slight tweaks on a main model, and we probably will have, like, tools that we can directly translate to these fine tuned models and maybe just do, like, a little bit of fine tuning on on on the tools that we have and the probes and such. To the question about thinking about continual learning and in context learning, my personal view... I mean, again, I I don't have strong perspectives around their... Around the architecture argument, but I think there's sort of a bit of a a bitter lesson around, like, prompting and in context learning and context engineering and harnesses and all that that they might be all you need for quite a lot of stuff. And I have a bias towards this because this is what I work on, but I kind of... For me, the most likely outcome in the... In any short term is that we don't have custom models that are really trained on very different data, but they're actually... What's happening is it's different context. Right? As you're saying, this is sort of what already happens. We have different LLMs that have different memories. And the memory is just the way I see it, a bunch of data that is tokens that are in context learned. Every time the model generates an answer, it it sees this it sees this long memory. And I think there... It's just... It's incredibly easy to get a lot of a lot of juice out of that without squeezing too hard, and then you end up with these memories that are pretty interpretable. And, yeah, I think I think some variables would have to change in the equation for me to see it being really that promising to fine tune models separately on individual people's data to have customization really pay off in that way. And so maybe my my my personal best guess is that we're gonna see more context engineering and more of this this kind of thing, more harnesses, more memories. And I think that's really what we've seen in the last year or so with with the rise of agents is is is a lot of a lot of that engineering is just going into, like, putting a lot of stuff in context.

[35:28] Nathan Labenz: Yeah. It's been striking that Anthropic has never had a fine tuning product, at least, that they've offered to the retail developer customer, and OpenAI has killed theirs. And what little test cases we do have around interpretability applied to different architectures have been, for me, like, quite pleasant surprises in terms of how well techniques have translated. I'm thinking, for example, of Othello GPT and then, like, Othello Mamba. Both were basically interpretable and seem to have, like, similar representations internally, and those were recovered through, like, fairly similar techniques. And, yeah, long way, those favorable trends continue. On the question of kind of change in models and, like, how much the the beasts themselves have morphed in front of our eyes over the last couple of years. I'm old enough to remember when temperature was a feature that we got to play with at the API level. And at that time, I knew how my tokens were being sampled from the distribution subject to, like, permanency. For all... There was a time... I'm also old enough to remember a time when I didn't know about GPU indeterminacy, and I was like, why would I turn my temperature to zero? Do I not get the same thing every time? And that turned out to be the answer for that. But I guess how how would you describe... Because it sure... The temperature's gone away.

[37:01] Eric Bigelow: It sure

[37:01] Nathan Labenz: seems like the models are... You know, mode collapse is maybe a little bit strong, but, like, they're quite overfit in some ways. Your story example is a good example of that. Like, GPT three with its wild, untamed nature was probably in some ways a better bedtime story generator than GPT six, which, like, always gives you the same kid and dog and waterfall. Even seeing that right now weirdly across providers where... I won't bore you with it. Regular listeners have heard me talk many times about how I use models to draft the intro essays for the podcast. But in testing Astra versus Fable recently, I've also noticed, like, very, like, uncannily similar

[37:45] Eric Bigelow: decisions

[37:46] Nathan Labenz: in terms of how they choose to try to present an episode. And I'm like, this is not random that they're coming up with these very similar ideas. There's some, like, pretty tight fitting going on.

[38:01] Eric Bigelow: And

[38:02] Nathan Labenz: then we also have seen, like, the introduction or at least the emergence, maybe is a better term, of these metacognitive behaviors where you see, oh, I think I'm off on the wrong track. Let me go back and approach this another way, the moments kind of thing. So how would you describe... How would you, like, tell the story of two years ago when you were doing this, like, super high volume sampling and you were seeing these, like, pivotal tokens, how have the underlying models changed and what should we be aware of as users and as people trying to understand how decisions get made and as interpretability researchers based on just how dialed in they seem to be. And I'm also just interested. Do you even know, like, how sampling is being done these days? It's a very in the weeds question. But, yeah, tell me the story of the last two years. How do you understand how models have changed during that time?

[38:55] Eric Bigelow: I'll say first that I wish I knew what the what the sampling parameters were for the cloud API and and for the latest GPT models. I think it's it's very unfortunate that they've talked these away. I I expect that it's probably similar things happening. It's just that they wanna limit how much people can reverse engineer the models and play with temperature to do a lot of systematic. Like, I think a lot of the people that are doing really systematic prompting are using, like, a lot of, you know, hitting the cloud API with millions and millions and billions even of token requests. There's a high risk that they're doing something, you know, a little bit nefarious or trying to distill models or or extract model weights in some way or another. So I think that's probably the main reason why they did this rather than that they're actually doing a different a different sampling scheme under the surface. Although, if I... I would love to learn more, and I'd be happy to be surprised. As for the story for the last couple years, so to your point that the earlier models were more, like, unleashed as you put it. And I think with the the word that comes to mind with this is something people have studied around output diversity, where what diversity means is that if you... It's basically the same... If you give the model the same prompt and you have it generate a bunch of different outputs, how many different outputs will it generate? And I think this is something that if you if you... There's been a lot of work showing that this disappears to some extent with supervised fine tuning, and the more the the more you post train models, the less diverse they... Their outputs become, which really isn't surprising at some level because the earlier, like, completion models that weren't even chat, they were just raw completions. Like, when you talk to it, it wouldn't even know that it was in a conversation. It was just doing text complete... Next word text completion on the whole Internet. It felt very much just like this raw thing that you're interacting with, this, like, big amorphous blob of Internet data. And if you just start giving it random tokens, it'll just complete those in some way that's coherent. Whereas afterwards, I think we evolved the... We first went into a world of supervised fine tuning and RLHF and other post training for chat and basically model... In instruction tuning, right, is really fine tuning models to this distribution of you're gonna be talking to your user who's gonna give instructions and you should respond to those sort of like a chatbot should. And then RLHF, I see, is further, like, sharpening that distribution around chat. You should act like a chatbot and talk to people in a coherent way that's co... That's consistent with this, like, chat interface that everybody is using now. And then I think RL does something very pretty pretty different, and I think RL can... I think I think a couple projects I've worked on have shaped a lot how I've thought about RL as well as some of these things recently about actually looking at the reasoning of models and seeing how uninterpretable they are. And I think RL kind of... I don't know how to think about it in terms of of it. Like, I think it it sort of sharpens the distribution in some ways, or, like, there's some really r l'd smaller open source models that are, like, kinda cooked. Like, they'll kind of, like, you know, I've I've I've done some... A bunch of experiments recently with this QUEN 3.635 b a a three b, like, model, and it it was... Are all pretty heavily. And when you give it the... When you give it this kind of prompt of, like, give me a story, two thirds of the story are about a clockmaker and a bunch of the other ones. I forget what it is, but there's like a another pattern that just comes so much in the stories that it generates. But I think, like, I think... I've actually tried this recently with like Astro for example, and it actually generates like, a bunch of different stories, which is which is interesting. Like, it'll actually generate fairly different stories with fairly different arcs and characters. I mean, I haven't done, like, a really massive scale of sampling, like, hundreds or thousands of different outputs and looking them looking at them, I I probably ought to.

[42:48] Eric Bigelow: But the impression I have of reasoning is that reasoning is it kind of shifts the distribution of what language models are doing in a totally different direction that's very different from, like, the human text that they were trained on, where they... They'll... Going back to this this phrase you said earlier that I really... I think very consistent with my thinking that they do, a linearized tree search. The... They'll enumerate, like, all these different possibilities of how the problem could be solved. And sometimes the possibilities that they enumerate are related to each other. But when... There's this idea of wait wait being, like, a really, like, key point where where the model changes its mind. But if you look at, like, deep seek model out... Like, r one's outputs, they'll say wait 50 times or more in a single reasoning chain. That's a lot of epiphanies to have when you're working through one problem. And I think this was really, like... This is a solution to this problem that models were hitting at the time, which was how can they backtrack, which is related to this forking I mentioned earlier. Whereas it's as it's generating a reasoning chain, it can randomly sample some word, and then it can't really step back from that path. It's just pushed itself down that path forever. But I think what reasoning models do is they get around that backtracking by just enumerating, like, every possible thing they can think of, and then later going back and choosing from that. And I think my impression of at least, like, probably the most state of the art proprietary models, like g p d six Astra, why can it generate different stories, is probably that it just generate a bunch of different possibilities in its reasoning chain, and then it's going back and looking at them. And that's where the sampling is really happening. I think this... I think maybe some of the diversity is happening within the reasoning chain itself is my way of thinking about it. But then, I don't know, I should probably do more systematic studies with RL models to to understand this this problem of of diversity better. But I just... I think... Yeah. Some recent projects have shaped where I really don't... I think reasoning is almost like a misnomer for what reasoning models are doing. I think traditional chain of thought reasoning really was like human reasoning in some way where it's like, let's break it down step by step and here's a set of propositions a, therefore b, therefore c, therefore d. But reasoning model will just enumerate a b c d a e f g f g a, all these different... It's like a... It's almost like a soup of tokens they create, and then they just go back and skim out something out of that soup. And, yeah, I think I think that's my high level summary of a lot of the research that's happened in the last couple years. And I personally... I think that in terms of raw capabilities, I think there's been, like, maybe some gradual change with this and a bit of extra scaling in the last few years. But then I think a lot of the raw capabilities were probably in these pre trained models the whole time. And a lot of what we've done since then is finding ways to just really evoke these and get these out and to sharpen the distribution of what models are doing. Coding agents, for example, is something where I think that just had to train models a lot doing coding to get them to behave in the right way even if they had learned the right things from the entire Internet Internet of text. It's like this extra version of this this additional case of this base models being just next word completion over the whole Internet. And I think a lot of the agent fine tuning and getting success with agents is really polishing what distribution they're fit to. You're a coding agent. You're doing lots of changes. You're working with git and diffs all the time, and you're running tests. And I think this is more than anything like a a small distribution shift or a sharpening of a distribution.

[46:41] Nathan Labenz: When you do research, how do you choose what model to work with? Obviously, a lot of people are choosing based on what the compute resources they have available support. I'm sure that's a factor for you too. But at Goodfire, you guys have significant resources, and you've built some infrastructure to do interpretability on whatever... Kimi k three. Right? Is it a frontier scale? Is it near frontier scale? It's big anyway. How... I guess, what is your mental model for what kinds of models to choose when? When you do something on a frontier Chinese model, in your mind, is that essentially synonymous with doing it on a American proprietary model? How much do you think of those as standing in for each other versus how idiosyncratic do you think they might be?

[47:36] Eric Bigelow: So I think I think of them as in large part standing in for each other. There are idiosyncrasies of, like, Quinn really likes to think in Chinese. And if you do the logit lens on Quinn, sometimes there will be a lot of Chinese characters sort of in the in the middle before it starts out putting English tokens if you're talking to it in English. For your question of choosing their own model that I work on, it is quite a privilege to have, like, these, like, big compute resources at Goodfire and something that I'm, I think, still still getting used to and finding ways to really, like, take advantage of in the best way possible. But there still is a problem of if we're doing, like, a lot of experiments, you know, I I think... It's still would be hard to justify me taking up hundreds of GPUs for a substantial amount of time to do, like, really, really elaborate experiments on k three if they weren't really, like, in line with some of the company's goals or sort of taking advantage of some time whether there's a bit of downtime and they're underutilized. So I think there still is. I I still do think about compute and kind of having some trade off of, you know, what's the right model scale where I can study the behaviors that are interesting enough. So I think generally, how I choose is I pick a model. Like, I I think there's... If you go down too small, then you're working with something that, I don't know, is is qualitatively different, I think, if it's, like, you know, in the below a billion parameters. Depending on the task, sometimes there there are still interesting behaviors you can get out. But But then I think there's a lot you can learn by, like, training models from scratch where you train, like, toy neural networks or transformers on on a version of your task and you look at learning dynamics where you do interpretability on them. But a standard that I've always set for myself in my research is to try to study something that's kind of like the big models where you do see these really elaborate human like behaviors, and you can just study them zero shot in the model without having to train the model to do that and without having to use, like, a facsimile of the model. And so with choosing a model, I usually... I I tend towards, like, the scale of seven to 8,000,000,000 or more being a sweet spot where, like, you can start to do good interpretability research. And I think also, like, the few billion models can be good if your task is is, like, relatively simple. But I think sort of around that scale of, like, 8,000,000,000 or more is when you can start to do this sort of, like, zero shot prompting and get interesting behavior. Like, my my story belief updating paper, initially, we just used a lot of 8,000,000,000. And I wasn't totally sure at first how it would do, but it actually does, you know, update belief very dynamically as it's reading stories across, like, a number of different emotions and other and other attributes that we tested. And so I've been I've been pretty impressed with models abilities of that scale. And then I mentioned that recently studying this QUEN 35 b a three b model, and I was also... I I was very very... When I when I started... Before working on the model too much, I wanted to just chat with it to get a sense of what it's like. And I wasn't sure if it would be kinda like, yeah, kinda kinda cooked as I said, like, some of the initial experiences with having it generate stories and seeing, like, a lot of the same stories. But then something that I I just like doing with LLMs in general is talking to them about consciousness and see what see what they say and how the conversation goes. And I think that that gives me some signal. Maybe it's, like, a strong bias here, but of how well a a model can, you know, think about a a pretty nuanced and complicated topic.

[50:51] Eric Bigelow: And I don't know. I was very impressed with how well this model was able to talk about the problem of consciousness and LLMs and sort of think actively about that. I mean, there's also this well documented... Or it's well documented, but also a bit speculated around what the post training data of these models is and if they're really, like, to some extent, distilling some of the some of the proprietary models. And to whatever extent that is true, that could explain why there's a lot of similarity with these models. But I think... Yeah. The... In terms of their core capabilities, they're all really trained on the same data, which is the entire Internet. And the main difference, I think, between all of these models is the post training and what is that extra special sauce that we're giving the models. And I think in terms of the moat between closed source, I I personally think that's probably been the biggest moat and will continue to be is is the post training that they're doing and at at various levels. And I think that that is one of the things that really differentiates the proprietary models from the open source models. And and I think what that post training does is it it will make the models way better at, like, coding or specific things that you do care about when you're actually using the models, but I think a lot of the core capabilities will be very similar. And and Kimi k three is also this big... I think this big step function in having... In terms of having a model that's really good at coding that we can study. And although I I haven't studied this personally, I've I've seen some of the other research that people at at Goodfire have done recently, like, with reward hacking. And it... It's it's... I mean, it's very impressive how good the model is at coding, and also it exhibits this, you know, reward hacking, like, crazy behavior that you just don't see at smaller scales, and I think does give us some signal in the studying phenomena that we think are happening inside proprietary frontier models. So I think, like, there... There's definitely a a a gap in performance between the proprietary models, the sort of, you know, o... Opening eye on anthropic models, and almost everybody else with a few exceptions. But I think, like, in terms of the kind of work we do with interpretability, this doesn't... I I think it's not too much of a blocker. The the main blocker is when there are certain behaviors that you just really can't study except with enough scale or post training. Like, I think, like, this reward hacking with with Kimiya is is like a really interesting example where it seems like just having the model pass that step function just in terms of some some particular behavior coding ability was very important to start also getting to see these really... These dangerous behaviors that we wanna study and understand better that, like, it's it's really hard to get it out of a smaller model. Or if you get it out of a smaller model, it just... It may be that it's happening through a very different means.

[54:06] Nathan Labenz: Does that mean that interpretability research today outside of proprietary model companies is becoming dependent on Chinese models? Is there an American model that you could use as a substitute for Kimi k three that you feel like wouldn't lose that much of the value, or are we... Like, we have to use Chinese models if we wanna do frontier interpretability in today's world?

[54:34] Eric Bigelow: I think for now, we do. And I think in the biggest models, I think there is this... There is a a a big gap particularly in programming ability between Kim and k three and everything else. But I think this isn't a gap that couldn't be closed easily if there was more investment in open source models from from American companies. And I think when you go to, like, smaller scales, the gap probably gets a bit smaller. Like, I think... I I I do study the Quinn models pretty heavily, and I think they are, like, really, like, state of the art for their model scales, and I'm and I'm very glad that they exist and I can study them. But I don't... I also think that if it... If I was stopped from using Quinn on the 35,000,000,000 scale and had to study the JAMA models instead, it probably wouldn't slow me down too much. Maybe a bit. I think right now, the gap really exists at the at the top of the food chain. But then, like, yeah, I think if there ever was, like, a ban on using Chinese models, I would really hope that some American companies would step up to the plate. Although, I think they'd have to have a particular business proposition in order to make that happen and in order to get the money that that they need in order to to train and deliver these models. But I would I would hope that it would happen. I think there is, like... There there have been some loose discussions around, like, placing restrictions on open source models bar none, you know, regardless of of where they're coming from. And I think that would be the bigger... That would be a real problem. I think that would really get in the way of a lot of good interpretability research. And perhaps, like, I think some of the most foundational research, I think, could really continue, and we could probably... If we... Well, depends what we had access to. But if we had access to some open source models, I think, like, there's a lot of foundational interpretability research that could have been studying LLMs from a few years ago and could keep doing that for, like, an... I don't know, a long time from now and continue generating, like, really deep insightful things and on helping us understand great mysteries right now. But then I think maybe some of the most important behaviors to study for risk, reward hacking, or some of the... Some of these sketchy things that can happen around misalignment that really are emergent behaviors with scale that only come out once you reach some some point of capabilities where it's... You're studying the same behavior and not something that's just like it in some way, I think those would would be slowed down if there was a a ban on open source models. And I I think that would... I I I really hope that doesn't happen.

[57:11] Nathan Labenz: Yeah. I'd hate to see you have to report to some bunker in the Nevada Desert to do your interpretability research underground or whatever the scheme would be in that scenario. One thing that I thought was really interesting in looking into your background is you just finished a PhD not too long ago. Congratulations. And it came from the Harvard psychology department.

[57:38] Eric Bigelow: Your

[57:38] Nathan Labenz: thesis toward a cognitive science of large language models. I'd say one of the biggest surprises for me in recent times has been how productive it has been to anthropomorphize models. What is your kind of personal philosophy of how to anthropomorphize? It seems like it's... We've... We should be backing off of a strong allergic reaction to it, but still it's obviously something we've gotta be using very carefully. So how do you use it carefully?

[58:10] Eric Bigelow: So, yeah, I really like that question. I think something that changed a lot over the course of my PhD... Mean, when I started my PhD, LLMs didn't exist. And I mean, my my adviser did a little introduction for me with my defense where he said, you know, if you had told me when I was starting out that this is what my thesis was gonna be titled, he was... He would say, oh, what now? What are large language models? And for me, yeah, when when large language models hit the scene, it really was just a drop everything and study these kind of moment. I mean, I had a colleague that recommended iSplit, like, specifically said that to me, and I think that echoed my sentiment at the time. I did have to, I think, push back earlier on. I did get a lot of pushback from some of the scientists around me who were used to the deep learning sort of being... There there being, like, lots of, like, shortcut type things where a model can't really do whatever behavior you're talking about. Even when it comes to, like, computer vision models can. Is ImageNet really computer vision? There's a lot of thing that... Things that vision means that isn't captured by... You know, take this, like, matrix of RGB inputs and give me a one out of a thousand categorical label. And so I think initially there was there was a lot of pushback on anthropomorphization. I forgot what the metaphor you used was, but I think it was some psychologists and and people in related fields like linguistics or or philosophy will sometimes look upon upon research on AI critically, and that it feels like, for example, psychology. Like, there's a lot of work around AI that feels like it's doing bad psychology or people use these words that have a lot of history in psychology and have been studied pretty deeply. And there's, like, these really fundamental theoretical and empirical problems that people thought a lot about, and people are just completely missing that when they're using these words. And so I think I understand.

[1:00:16] Nathan Labenz: Could you give an example of that? Where... Like, where do people go real raw?

[1:00:21] Eric Bigelow: I think, like, for a while, there is resistant on... I I I think there is some resistance on, like, talking about, like, planning in language models, and the language models really plan things out. And even around, like, having beliefs or knowledge of of things, it was sort of, like, they don't really know stuff. They are kind of like, maybe they're not quite stochastic parrots, but it's sort of something like that. They're just doing a lot of next word prediction and this kind of being a a little bit of a dirty word that like what humans are doing is probably a lot more complicated. But I think over the years, I think some of this pushback has softened and some of that just has been how damn successful these models have been. Right? How everything... You know, they've they've it's really taken the world by storm in an incredibly dramatic way. And at some point, you can't keep pushing upon... Pushing back on a river that's just pushing so so hard. You know, I think... I I... Something that I really want more people who... In these fields who are critical of the research do is instead of just seeing it as problematic when people use these words to come and, you know, share with... Share share your own perspective. You know? Start talking to interpretability researchers or whoever who's studying AI and start talking about, like, what are beliefs? What are decisions and models? How can we understand these better? And so I think broadly, there's two perspectives I have on anthropomorphizing AI, where one of them I think is that... I think you can be... Like, there's some level at which you can anthropomorphize. That's kind of like, you don't have to be too careful about it. It's for you. It's for you to understand it. You don't have to take the words you're using too seriously. I mean, if my car starts making, you know, a clunky sound as I'm driving, I might say that it, like, it's trying to tell me something. I mean, I think this is a little bit dramatic example because, obviously, my car is not trying to tell me something, but it can be helpful to think about things that way. And there's a... There's this word that's used in in some parts of of cognitive science on intuitive theories. And I think about some of this being... Like, an intuitive theory is these these frameworks we have for thinking that are largely ingrained by, like, evolution that are higher level than just, like, really, like, you know, basic perceptual things. They're like systems we have for reasoning about phenomena. Like, for example, like, reasoning about people, we have a lot of machinery for. And it's very easy for people to intuitively reason about, like, agents as such. And so sometimes it helps me to anthropomorphize my car as an agent and the problems, the sounds that I'm carrying as it's trying to tell me this. And I think at that level, it's... As long as it's helpful, by all means, do it. And it's just... I think we kind of go into this interesting territory though with LLMs where the behaviors they're doing are not just making clunking sounds. They're like... They're doing... They're they're speaking language. They're they're doing mathematical reasoning. They're doing social reasoning for that matter. They're helping us answer scientific questions. They're doing programming.

[1:03:58] Eric Bigelow: And so the other perspective I have on anthropomorphization is that I think I think what's really healthy for everybody, the the people, you know, from these fields who might be... Who might have a critical eye of, like, words being, you know, misused or wheels being reinvented is to try to take those questions seriously. What are beliefs or what are decisions for that matter? I think... As a disclaimer, by the way, I'm not an expert on human decision making, but thinking about this project has gotten me a lot very excited thinking about all these things with people too. I mean, I think this this notion of, like, of forking paths and sort of whether like one token being generated could send you down in a very different path. It's... Initially, my thought was this is like so different from what people do. Like, I have this example in the paper that if a if a person was like there... If a person was speaking and they misspoke, if they said, you know, Billy fell into a hole or Billy fell into a spaceship, depending on what word you see, you might... If you misspoke and you said a hole in I mean, I think I had a better example in the paper. But if you if you misspoke, you would correct yourself and you would just keep telling whatever story you're going to tell. But for an LLM, it might just commit to that path and start saying this very very different story. So it seems very different. But then I think there's a lot of things in life that actually are like these forking paths where actually you do kind of just flip a coin and depending on what happens at that step, that's what you go with. And sometimes we don't even see it that way. I mean, there's a bunch of work around rationalization of decisions where... Like, for example, there's these cases where you have people choose from a a set of stimuli which one's better in some way, and then you later give them the stimuli and you ask them to explain their responses. But as the experimenter, you might change the stimuli around. So you might actually give them different ones than they actually chose, and people come up with explanations as why they chose it. And so they'll rationalize decisions that they didn't even make. And so just to go back to this point, think there's, like... I I I think there's a really, really fruitful direction, which is to try to really take seriously the terms we're using when we are anthropomorphizing and try to study these better. Try to, like, I think when people from fields like, you know, psychology, neuroscience, linguistics, philosophy are are critical, it's it's often for... Some... Sometimes it's not for a good reason. Sometimes people are a little bit afraid of, you know, losing ground on something that is, like, very important and that there... You don't want people... A whole field to be dominated by low quality work. But often, it's coming from a place that there's, like, really important theoretical and empirical problems that people have thought a lot about how to solve. And I think the more we can, like, you know, get what we can from this previous work and instead of reinventing wheels, start building, you know, chassis and car frames on top of the wheels we already have. I think, like, that's a future I wanna go towards. And I think it... It's just... I think a lot of a a lot of people who are critical in in in cognitive science, for example, which is more my field of... The the field I'm most familiar with. I think I think LLMs are so interesting. I think a lot of cognitive scientists, at least in... Until recently, thought that they were boring. It's it's stochastic parents or the models are doing something dumb. But I I think that this is, like, a real opportunity to learn something about ourselves and how do we even think about this thorny theoretical problem. If we might say, well, models don't have goals or intentions. Well, like, what does this mean? I think we we take this a bit for granted in our own field because we study people and animals and other... And biological organisms that seem to evidently by our own framework of defining these terms have goals and intentions. But I think it's a... I I think it's an opportunity to to breathe new life into these, like, deep questions. And I think there's people that are really excited about these problems and also a chance to make a really big difference by answering them.

[1:07:36] Nathan Labenz: So if we look at beliefs, for example,

[1:07:40] Eric Bigelow: you

[1:07:41] Nathan Labenz: have looked at a bunch of different... What is a belief is one maybe one question we can hold in mind as we talk about some of your techniques. You have done this sampling project to figure out what's the the full universe of possibility that an AI might generate in response to a given prompt. If there is a question where the task is to get the right answer, then it seems like you're making an argument that at these pivotal tokens, the belief state changes.

[1:08:18] Eric Bigelow: You've

[1:08:18] Nathan Labenz: also got, like, low

[1:08:19] Eric Bigelow: sample

[1:08:19] Nathan Labenz: techniques with some tricks that allow you to do that more efficiently. And then you've also looked at internal representations and tried to predict what the what the result is gonna be based on those internal representations. What does all that tell you about what is a belief in an LM? Do you have an account for what a belief is? Is it simply reduced to the probability that it's ultimately gonna give answer and those shifting probabilities? Is there something more... That doesn't feel like it captures the fullness of what I think when I talk about myself having beliefs. Is there something more that you would talk about when you think about an AI having belief?

[1:09:00] Eric Bigelow: Yeah. I think it's a it's a great question. I think I think there's a couple a couple of thoughts that come to mind with that. One of them is that I think there's some level that it does actually relate to sort of simple things, like the probabilities of tokens that a model will generate, except there's more nuance to that. There's a question of, like, how does this change across a lot of different inputs? Or how do these beliefs change through the course of in context learning, or as a as a model's reading a passage, or as it's generating a a chain of reasoning, how do its beliefs and different and different concepts change? And some of the mental framework I have for beliefs is around assigning probabilities according to latent hypotheses or concepts that the model has, and then its behavior is generated according to latent concepts, and the latent concepts are which which concepts are evoked is is really determined in large part by the data that it's given. And I think this this really speaks to a a framework of cognitive science that I'm very fond of and and has influenced me a lot, which is around Bayesian modeling of cognition, where a lot of cognition can be thought of as Bayesian inference at some level where people have these latent concepts, and we have prior probabilities of bringing these concepts to bear on sort of regardless of what the context is. And then context or input data will lead us to reweight these... Our beliefs in these latent concepts through posterior inference. And by, you know, choosing which concepts or... I shouldn't say choosing, but by by by doing inference and this inference selecting different concepts over one another, this this can explain a how a lot of how human learning works and how humans can adapt their behavior on the fly to lots and lots of different situations. And I think this also applies to LMs as well. I think something that this I think a bigger picture thing that that this question of beliefs also evokes though is, like, what is what is being represented in an individual, like a person or an LM for that matter, when it believes something? And I think this is room for for, I think, cognitive science to also grow and and maybe do experiments that we never could with people where the Bayesian framework in cognitive science sometimes has it it it... A criticism of it is that sometimes the the claims that it makes are... It's unclear whether these are really saying that people are doing a certain kind of inference in their brain or how this is implemented. What is the algorithm underlying this behavior or this computation? And sometimes these views are, like, agnostic to that. And it's easy to be agnostic when it's just really, really hard to study brains, and it's really, really hard to uncover what truly are those algorithms or, like, how are things implemented. But we don't have that excuse with LLMs. It's very very easy to peek inside the brains and to do interventions, which is very complicated and both for methodological and ethical reasons with people. But we can do them with LLMs, and I think it's... Gives us a chance to actually break new ground around understanding, like, how beliefs are represented and when models are doing inference. Are they drawing samples? Are they estimating posteriors in in some of the ways that we've used to model people? And are those models really what's happening, or do they just describe the aggregate behavior as what's happening under the surface, really? Does it defy these models, and are these models really just displaying the behavior instead of the algorithm? I think another another point to your question is around this way of, like, the the forking past methodology of, like, resampling all of these rollouts at every point in reasoning, we use this word in the paper, which is uncertainty. And I think one of the things I wanted to do with this paper is just highlight how strange it is to think about certainty with reasoning. Because, like, by the end of a reasoning chain, the model is almost... Because you... Like, at least back then, they were almost a 100% sure what the answer is because you basically... Whether the... If you're proving some mathematical theorem and you're trying to say, is it... Here are these different proof outcomes you could have, you end up with a long proof. Maybe it's wrong, but it leads you to one conclusion.

[1:12:41] Eric Bigelow: And so it's... The model is not certain at that point or has no uncertainty about what the final answer is. It's determined by the previous reasoning chain. And so what we describe uncertainty as in reasoning is you resample these long reasoning chains. And so say at this point of reasoning, I resample and I get 50% answer a and 50% answer b. We say that the model is fifty fifty uncertain between a and b. But that doesn't mean that it really represents these full rollouts. That means that if you were to do reasoning from that point on, you would get fifty fifty. And so something I've been exploring with some recent work is is an... Is better understanding, like, what do models represent? And it... And and it's actually pretty hard to just decode these outcome distributions, these rollout distributions from the hidden weights. This... A disclaimer is this could be a... An issue with just not having enough data, and this... These methods are just so expensive in terms of inference that it is hard to collect a lot of data. So this could be a data issue. But an... Another explanation is that models actually, when they're uncertain, when they're they're... They are sort of uncertain over reasoning, that they're not really representing the full reasoning chains. They don't really know where they will go next depending on which word they will choose. But maybe they represent just, like, a couple steps ahead. And I'm working on a project that's sort of studying exactly this and showing that there are... That you can actually can decode to some extent what models are going to do next, and you can actually guide sampling based on this. And to some extent, you can guide sampling based on their predictions of their sort of final outcomes. But when it comes to, like, really, like, this question of whether models are representing the full reasoning chain before they really generate it, I kinda don't think so. But I think there's really an open question of how much they do represent. Like, when when a model could go through one reasoning chain that leads to a and one reasoning chain that leads to b or generate one story or another, does it know the full story that it might generate? I mean, I I don't think it knows the full story, but I think a a a neat connection I make with... Or I I see their being with with theories of, like, human cognition is around how people can... Like like there's this stuff around mathematicians and how mathematicians, when they're like starting to solve a proof, they... Of course, they'll like take... It might take them a really really long time to get a full derivation, but sometimes they'll have some like high level scaffolding view in their mind. They'll be able to see the shape of the proof even before they complete it. Like, they haven't actually gotten through all these steps that get them from point a to point z, but they have some high level representation of what is the shape of that distribution. And I kind of wonder if that might be true for LMs too, that they actually have... That they might not represent the full story or the full reasoning trace they're gonna gonna do, but they actually have some uncertainty over sort of the high level the high level paths they might take. In looking at some of the

[1:16:22] Nathan Labenz: graphs from a couple different papers, including the most recent one, Forking Fast, the the sort of sudden phase change is, like, an extremely striking result. How do you understand that? First of all, it's weird when it just would happen on a character like an open parenthesis. In general, do we feel like these phase changes happen at... And you've got this other paper about the sort of performative nature of at least some chain of thought. Do the phase changes tend to happen before or after? Or I'm sure it's some mix, but, like, you could imagine a a sort of rollout happening getting to the point of the conclusion of the thought and then the phase change happening, which would be kind of weird. But then it's also quite weird if the phase change happens at the beginning and then they have to roll out all these tokens already having had the phase change. To compound that confusion even further, I don't feel like I'm doing that. Sometimes I do. Occasionally, I feel like I have, like, a a eureka moment. More often, though, I feel like I'm edgeily landing the plane. And I do wonder to what degree we might see quite different things when we go to looped models as we now may have in production with Astra. Right? If you don't have to pick a token at every time stamp, then maybe you can gradually go toward a conclusion and maybe that's better. If something feels weird and like wrong about these like sudden token level phase changes, maybe not, but I want to see that like as cognition is happening, change is gradual and this would feel much better to me. I guess to make that a question, do you share that intuition that something is weird or wrong with these cliffs? Would you also like to see a gradual landing of the plane? What do we know about how these phase shifts relate to the relevant portions of rollouts, like before, after, or whatever? And should we expect it to be harder, easier, better, or worse as we go into thinking in latent space? Just to give you some easy ones.

[1:18:39] Eric Bigelow: Yeah. I I really like all these questions. I'll try to keep them in mind, but please remind me if I... If it slows my mind up. I think the question of whether I'm, like, bothered by them. I think a lot of people, when I first described this are are kind of, like, a little bit unsettled of, like, oh, if one token had been different, actually, I'd get a totally different output and how could we train me as a way? But I kinda... I think that if you really think about the problem of output diversity and if you want some diversity in what your model will do, you want it to do different things, and you want these to be self consistent, you don't want reasoning chains that are just nonsense, then I think forking might just be an inevitable property of having these two key ingredients. And what I'd really like is I'd really like if our user interfaces surface this to us, if we had some level of this kind of, like, uncertainty, if you call it that, surface to the users. I mean, I mentioned earlier that there are these inner... Earlier interfaces of showing you the token log probabilities. And there's another kind of interface called a loom developed by by a person named Janice who did some some stuff that influential... Influenced this work. And I'd I'd really like to see something like that be part more part of the standard user interfaces that we use for interacting with all m's except at the level of, like, semantics. I wanna see the points where things could branch off and be a very different path highlighted to me as I'm interacting with the all... I think if it's... Whether it's gradual or it's sharp, if there is this kind of, like, change that's happening and you could get very different answers by resampling, and maybe, like, the distribution of different answers is more similar at the beginning. If it's gradual, it's like, by the end of it, it actually might be a very different distribution than you had at the beginning. But I'd still... I think I'd really like to see that surfaced. And I think that's how that's how I think about this is like, instead of this being necessarily a a bad scary thing that can happen where there's like a sharp left turn and you didn't know it, I think my my personal hope would be that, like, whether there's a sharp left turn or a gradual turn left turn that you know it. That this has actually surfaced to the user in some way. For... Can you remind me your other two questions? Yeah. This... The kind of

[1:20:58] Nathan Labenz: mid one was how do the phase changes in in kind of rollout time, how do they relate to what would appear to be the relevant text?

[1:21:09] Eric Bigelow: Yeah. So I think at least for what I've looked at, it really comes in in all different... In all all these versions. It happens sometimes at the beginning of rollout, sometimes at the end, and sometimes at the middle. At least in that earlier paper with, again, a a model that's now a couple years old, there were quite... There were more at the beginning and the end of sequence, but this also changed depending on task. And there were lots of changes in the middle. I mean, I think I think one big takeaway I have from that is that really every every input behavior can tell its own story. There aren't always these dramatic uncertainty dynamics, but when there are, it's really, like, it really depends on what what input and what reasoning chain you're looking at that it really it really has its own story to tell. And and I think if you look under the surface, sometimes there is... There's lots of cases where it is... You can see that there was some step of reasoning that really could have been could have been different, and the model was uncertain about this particular step. And sometimes... I think the one thing that could be mitigated and maybe we would hope to train away a bit is some of this sharp forking happening at unexpected tokens where I think this, to some extent, speaks to, like, these sort of weird biases of LLMs that they have in text where, like, that open parenthesis token, there's just, like, a some strange bias in all of the human Internet. There's a lot of text that uses kilowatt hours and does parentheses, k w h, that just happens to correlate with something semantic in in some funny way. And I think maybe if there were fewer fewer of those, it would be it would be much better. And if this forking really happened at points that were interpretable for people, I think I think that would be better. But I think to what you said that there... Something you said that you thought it would be, like, a little bit unsettling if sometimes this... There were these changes, like, at the end of the sequence. And actually, I I I... You see these quite a lot when a model seems very uncertain about what the right answer is, and and it just doesn't know. And maybe it's like, if you do a bunch of rollouts, there's, like, one of the answers is more probable. But you see a a really common pattern where there'll be, like, stable uncertainty rate up until some point when it just suddenly collapses and it becomes suddenly certain. And and sometimes that is really when it finally says the answer is blank. And that really is just... It it was just really uncertain until that exact point. I mean, one thing... One kind of funny example of this from that earlier work was there was a there was a task called last letter that that was used for LMs at the time. I'm sure all the modern city yard LMs can do it no problem, but you take four words and you take the last letter of each and you concatenate them. And the model at the time had a really hard time with this. And and some of the reasoning chains are funny that it'll even write like Python code to solve this problem. And then it'll finally say, okay, the answer is... And then when you look at the alternate pass it could take, it really is just all these different final answers of of letters that are ran... Kind of sampled from within these words. And and really it is just at that final answer moment just kind of making it up on the fly. And, yeah, I think, like, I think surfacing some kind of understanding of certainty to users is something I really wish we we had more of in user interfaces. I mean, I think I think one of the big problems with this kind of forking as as we kind of tried to get it with this forking fast paper and also another project on this around decoding the... Decoding outcome distributions from hidden representations is that it's very expensive. It requires, like, a lot of sampling to get real uncertainty over reasoning. And so I think the forking fast paper... Our solution is to use some statistical models of this distribution where we model it as being smooth in some places and jumpy in others, which, like, is supported. If you draw even more samples, it becomes even more smooth except at these key points where it really suddenly jumps. But really the best... The most efficient way to do this has always been with hidden representations, if we can find some way to do that. And so, yeah, I would be... I'm still hopeful that we can have a world where there is really a really efficient uncertainty estimation with hidden representations. And although they think this is even more hopeful because most of the providers really have just converged onto this chat interface that feels, I don't know, it feels very... I think the chat interface is is kind of uninspired. It it feels like it it really hasn't changed from, like, the Eliza days of, like, you know, fifty years ago or more where it's really just text input, text output. And I think there's so much more we could be doing with user interfaces. You know, I think... I I really wish we could kinda move into, like, the GUI world instead of the bash terminal world of interacting with chat and just giving more information to users, but in a in a polished and coherent way.

[1:26:07] Nathan Labenz: Just the last couple days in the online arena, there's been a eulogy or whatever for the stochastic parrot meme. And in a way, you've... You're, like, saying it's maybe not entirely dead yet. I guess one one theory of this sort of regenerating tokens are, like, probabilities aren't really changing as we're generating tokens, and then we have a sudden jump at the end of this passage where now, like, sort of the die is cast. One interpretation of that would be like, you're missing something. There is something more going on under the hood where something is changing and it's gotta be more gradual. You just haven't found it yet. Another interpretation would be like, no. That really just remains uncertain till the end and then we make a random choice of a particular token and that is what actually leads to the difference in behavior. Can it be both in different cases, and does that leave some... Should we still have some space for the stochastic parrot?

[1:27:09] Eric Bigelow: I don't know. I I I think I I think the sarcastic parrot metaphor should probably be put to rest. I I I used it a bit earlier as this criticism coming from from earlier in my PhD from cognitive scientists and psychologists on feeling like LLMs weren't interesting to study. But I think it's it's really sort of a a fairly shallow argument that doesn't really say very much. I mean, I I think it's sort of like a Chinese room type argument where, like, there's a a giant book that has memorized examples. But I think it's like, what the heck is going on in that book that lets you respond with conversation? Like, whatever these parents are doing, it's very, very interesting. And however it is that it's like, what... I I I don't really think the idea that it's just like a giant lookup table holds much ground. And as... And I think there's also just so much work showing, like, consistent world models. I mean, you mentioned the Othello GPT, but I... There's so much work around, like, looking at the representations and and and LMs and showing that these are, like, really structured world models. And at some point, it's more efficient to learn a world model than to learn a giant lookup table. But then I do think stochasticity and sampling is still such a integral part of the equation, and it kind of has to be if we want some... If we want there to be, like, output diversity, if we want models to be able to do different things, if we want them to also have, like, uncertainty and to to... Like, maybe maybe there's a world where they just voice that uncertainty. But then if we don't want them to just halt their reasoning every time there isn't a 100% certainty, then there has to be a bit of this kind of, I think, flip of the coin and and following a path. Think the more that can be surfaced to the user and the more the user can know how uncertain was the model about this thing that it reported as a fact when it may have been a hallucination. The the... That's... The better it is if if people know that. But, yeah, I think stochasticity is is definitely not dead. And I was earlier you mentioned, like, this idea of loop transformers, and I think loop transformers in this world of, like, latent chain of thought that we might be going on to with with GPT Astra really seems like pushing the needle on that in a big way and making, especially in terms of efficiency and not just performance. I think this really is something that I'm spending a lot of time now, like, trying to just wrap my head around and think about this in relation to my mental model for how LLMs work. Because now reasoning can happen during this deterministic forward pass in some sense. I mean, maybe it's looped and maybe it has some amount of depth, but in some sense, it's really just deterministic. It's like they're... The... All the weights just do the computation. And... But something I've I've recently been been doing some, like, forking paths analysis on the the open source loop transformers that I can find. Like, there's the series of models called Oro, and you actually do have some interesting uncertainty dynamics. As the models are generating their tokens in their in their reasoning chain after doing this latent reasoning. It still is uncertain between answers, and it still does change at particular tokens. And so maybe the mental model shifts a little bit in terms of, like, all of the... How much of the thinking is just happening in that token stream. But then if the model has some uncertainty and if it, like, in its latent chain of thought explores the different possibilities, maybe it still happens. That as it's generating a response, it still is doing this flip of the coin next word prediction. And I don't know. May... I I think maybe that's a good thing. I I I... It certainly helps for my mental model, but I think this, like, I I don't wanna live in a world where you always get exactly the same output from a model if you give exactly the same input. That feels like if we're in the world where that's happening, then there is no diversity of perspective and we've just, like, really... That's like the most extreme version of mode collapse, you know, where, like, your one output will only give you the... Your one input will only give you the one output. And I I I don't think lane lane chain of thought will completely change that because you still eventually do go into this sampling tokens regime. But, yeah, it it might change the... It might change that in in in some pretty dramatic ways. And so, yeah, I'm still trying to, like, trying to wrap my head around that and trying to think about, like, what's going on inside the latent chain of thought and how do we how do we think about that with some of the some of the tools that we have and even just metaphors that we as scientists use for understanding the models?

[1:31:49] Nathan Labenz: Do you have intuitions for why the chain of thought is getting wonky? It's a quite different dialect or way of processing information clearly that goes on in the chain of thought with all this, like, vantage illusion... Illusions, disclaim, whatever. We also see this now with Astra and the way that it sends messages to its sub agents, which I had naively thought has to be some sort of, like, length penalty or brevity reward or something. But then it seems like maybe not because people are, like, counting tokens and saying, well, you could just say the thing, like, in fewer tokens if you just said it. So it remains a mystery as to... And I don't know if OpenAI is a better explanation, but they certainly have been saying, we're looking into this. It's important that we wanna figure out why this is happening. Any intuitions that you would offer that people might wanna take inspiration from, chase down, put into the the hopper at OpenAI for what the hell is going on there?

[1:32:52] Eric Bigelow: Yeah. I think I think that reasoning kinda presents this really interesting challenge of distribution shift where when... Or like RL in general, where when you're when you're doing RL, whether it's with, like, a single model or with sub agents, there isn't necessarily an incentive to keep things human readable. Right? Like, the objective is to find the... To have the final... If you're doing, like, outcome level verification instead of process level verification, then your objective is to basically have a better final answer by whatever means possible. And it kinda reminds me of, like, things with, like, language evolution with people where, like, if you look at a long enough time scale, like, human languages change. There isn't really, like, any driving force that forces the English language to stay exactly as it is today as it was three hundred years ago or six hundred years ago. And languages do change. I mean, there... There's, like, exceptions of this. Like, the the French government, as I understand, has an institution for sort of keeping the language as it as it is and has had that for a very long time, which has sort of resisted forces about language evolution. But this naturally does happen. And, I mean, there's also, like, I think microcosms of this that are studied sometimes in, like, communities that don't have language and language, and they need to, like, develop their own language, and you can see it evolve just over generations of people, not even, like, centuries. But then I think there's also a bit of the the sort of Goodhart's law thing of if you if you optimize for an objective, it's no longer it's no longer useful. And similarly, like, if you if you optimize for a chain of thought being monitorable, then sometimes the chains of thought are less faithful. And, like, what you end up with is a different objective function that now is prioritizing. I'm going to learn intermediate steps that look sensible, whether or not they are sensible. And I think, unfortunately, that's the kind of the that's kind of the situation where we're with... In with RL where we're kind of giving models like this freedom of, like, if we're only if we're only giving them reward or punishment based on their final outcome, then they might as well speak to their sub agents in, like, weird nonhuman languages. And then if we force them to speak in human languages, maybe they're using words in different ways than we expect them to where they're actually, like, they're bypassing the monitors in some way. I mean, I think my thinking on this has shifted pretty pretty dramatically in the last, like, year or so. Where I used to be very bullish around, like, chain of thought monitoring being promising, where even if it's not perfectly faithful, there's something about it that it it is strongly determining what the model can do. But then some of, like... There is, like, the stolen chain of thought paper that shows some of the chain of thought of GBT models, and they're, like, they're talking about, like, musicals in a way that feel... It feels almost like spy language where you're saying musical, but really, you mean something very nefarious. And there's lots of this happening. And I think it's kind of obvious in retrospect why this would happen because, again, if you're doing RL, you're letting models just shift the distribution in such a radical way from what they're trained on. So I don't know. I don't... I I think I still I still like that there is chain of thought and that things do get forced into a token stream even if it's harder to read. I'd like to think it's easier to understand, like, a LLM made up language and and probably really interesting. I mean, just scientifically, like, I kinda like... I really wanna understand how Astra talks to its sub agents. That's pretty cool. Like, if it's if it's not like just a length penalty forcing this, like, how did that change? And could we look at, like, generations of this and and understand that just scientifically? And it would probably also help us a lot for, like, monitoring and understanding these these new languages. But then, yeah, we might... Again, for this lane chain of thought point, we might move move into a world where, like, chain of thought is latent and how, like, planners talk to sub agents is also latent. I... There's no inherent reason why that has to be tokens. Again, I I I hope it does stay tokens. I think there's another layer of interpretability and, like, really interesting scientific problems to study there. But, yeah, I think there's no real real forcing function to keep things really nice and human interpretable. And if we add a forcing function, then maybe they're no longer human interpretable or or really meaning the same things that they used to.

[1:37:21] Nathan Labenz: So if I'm trying to square your comments just now about this quick dramatic evolution of language with enchanted thought and with an agent to agent. Your earlier thoughts on why continual learning won't make things, like, totally impossible to interpret. Is it just a question of scale? Is it like your model will be with you for continual learning for not that much time, you know, sort of human scale amount of time, and that's not enough for everything to get totally pushed out of the the representations that we can understand as baseline. Whereas these RL training runs are just so massive that it's essentially operating on geological time. And therefore, even if these, like, drifts are slow to accumulate, they just add up to something so strikingly different before and after. Is that the story you would tell, or would there be a different way of kind of synthesizing those two analyses?

[1:38:26] Eric Bigelow: So just to be clear, when you're talking about continual learning, you're talking about the world where we're actually, like, fine tuning models for each person. Right? Actually, like, changing their weights. Is that right?

[1:38:35] Nathan Labenz: Yeah. I think so. I don't know. Take your take your continual learning. But I'm just... Who knows? Right? Who knows what that will ultimately look like when it happens? But you were saying earlier that context always changes representations, and it probably doesn't change it too much. But it seems like something in this RL thing has changed them quite a bit. There's there's some, like, qualitatively, woah. That's, like, shockingly different than it was before. And I'm guessing it's just a matter of scale where you're Yeah. If it's just you and your agent for a while, sure. There might be some, like, perturbations or representations might get morphed a bit, but, like, you're only one person not gonna push it that much. Whereas, like, a zillion RL episodes cumulatively can make a big change. And so there you have it. Maybe no more explanation required.

[1:39:20] Eric Bigelow: Absolutely. So I think I I think this is a very this is a very interesting point that, like, maybe the representations actually are changing a lot during oral as these, like, languages themselves are changing. But I I I mean, I I think the only solution is to study study the models that we can't study, right, to study these models that are behind closed doors and actually track what's happening in terms of representations. But I think my my answer around continual learning was I I was talking about the internal representations of the model rather than the kinds of languages they speak. I mean, say that you have people that speak a very, very rare dialect of a language. There's only, like, 200 speakers or something like that. And maybe they're even trying to keep the language alive, and so they do some fine tuning of the model so they can speak their language. Even in that case, I I would expect most of the internal representations of the model about... It's about how it thinks about the world and, like, all these thing... All these, like, internal world models that those have developed in order to, like, operate effectively. I'd expect that, like, a lot of those are staying the same, but the the sort of surface level mappings are changing. Right? Like, there's these classic things about, like, color spaces and how different languages have color spaces. Right? And I would bet that as a model learns different languages, color space. I mean, the the funny thing is that the language might actually reflect there being different cognitive biases of the people, which might actually represent a little bit of a shift in the world model itself. But I'd expect a lot of it to be, like, input output mapping changes, that it's kind of like learning a new language, but a lot of the internal representations are pretty similar. But then... Yeah. I mean, it's honestly... It's a it's a great question that I would just love to be able to study myself, like, to be able to really know how our representation's changing over the course of RL. And when when models, like, if you can measure how weird their language is getting, like, what's happening under the surface, is there really, like... Are there really deeper changes happening? Are there, like... As the sub agents go into these, really compressed pseudo languages, are they developing, like, are the internal representations shifting in a dramatic way to be different? Maybe, maybe not. I'd I'd really love to know.

[1:41:31] Nathan Labenz: Yeah. Yeah. It's a great point that the... And there... I think there is quite a bit of evidence to support the idea that the internal representations are highly conserved even across a bunch of different languages, even languages across different language families, etcetera, etcetera. So that is a a quite good observation for that question. So where do we go from here? We're we're at this moment where we're in a rush to hand off a lot of, let's say, proximal control over all sorts of important systems to AIs, we're gonna them in language, then they'll do all the actual work. And we don't have a great account of... Of course, there's, like, the sampling process on a distribution, but when we get to these critical tokens, it seems like we still don't have all that great of an account of how does that distribution from which we then sample get determined?

[1:42:34] Eric Bigelow: How are we gonna go get it? Well, I mean, we just need more interpretability research. I think it was just the... It's it's it's the conclusion I've I've come to for quite a while now. I mean, I I think we're, like, racing to come up with, like, theoretical frameworks and empirical tools that can keep up with the models. But for the amount of investment that's happening in making models better versus the amount of investment in understanding them, it's just... It's no comparison. You know? There's, like, trillion dollar companies that are... That that are producing these models, and AI as a whole is just, you know, many trillions of dollars. And I'm in, like, the what... Like, the largest interpretability company, maybe maybe the only substantial one that's... That has a value of over a billion dollars, and I feel like this is a big deal. Right? A billion dollars is a lot for a company, but it it... There just is not, I think, enough investment here. And and not just in industry, but also in academia. I mean, I think there's, like, a lot of great research that's happening in academia. And academia is always... It was, like, criminally underfunded. And, unfortunately, given at least in US, the current the current state with the government and funding for academia, it's it's getting harder. And and so I really like... I'd like to think that this could largely be solved with resources and devoting more resources to this and time. And I think, like, yeah, I think we're basically just, yeah, racing racing to keep up with this and understand the models. And and just to be clear, I think when I say interpretability, don't... I also don't want people to feel that I'm, like, excluding, like, evaluations because I think it's all sort of, like, one big spectrum of understanding AI. And the the work I described earlier with understanding chain of thought and doing this resampling really is, like, in in principle, like, behavioral. And when you see these, like, in context learning dynamics, you can see that with behavior. I mean, it's also really critical to look in representations so that you can do things, so that you can know what's happening under the surface, which may not be obvious, so you can do things more efficiently. And and, like, I think these two points of, like, looking under the surface and doing things efficiently are, like, just huge. Right? Like, if you wanna do things at scale, you want to do things efficiently. You can't you can't do things inefficiently when it's happening. Every single person's machine or, like, every time you're talking to an LLM. And so, like, I think we really need more tools at scale. We need better theoretical foundations for for putting all these theories together and really understanding, like, what's going on in an LM from soup to nuts. And we need a better grasp so that we can answer the important high level questions that are now, like, so critical as we are just, you know, handing over control of important levers of power to AI and, you know, pushing lots of cognitive labor into them. And even people, like like, students using them to, like, to, like, learn learn about a subject and offloading some of their own cognitive effort, I think, like, yeah, we're... I I I I just think that, like, understanding AI is, in my opinion, it's just the most important and one of the absolute most interesting topics that you could be working on right now. So that... That's what I that's what I hope we get more of is more understanding of AI, whatever that means.

[1:45:57] Nathan Labenz: Any tips for Silico users? The the obvious... Yeah. I think this whole thing is a little bit farcical at times, but, obviously, all else equal, we might as well bring the agents to bear on this interpretability challenge. It's been too long since I talked to Dan about it as as Silico was just launching Bill. Have you guys learned broadly? What have you learned? What should I be sure to take to my Silico sessions to get the most from my my precious research units?

[1:46:24] Eric Bigelow: Absolutely. So so the thoughts I have here, think, also apply to other agents too and not and not just Silico. I think it can be really tempting to try to just, like, hand over an entire project to to agents and just say, you know, figure out this big open ended problem and run it... Let it run a whole bunch of experiments and see a bunch of plots that come out and a bunch of, like, descriptions. But it's really easy for, I think, agents to kind of, like, get a bit, like, lost in the sauce and kind of, like, create these theories that aren't fully formed or don't fully make sense. And, like, I think, like, the best way to use agents is to give, like, a little bit of hand holding and kind of, like, keep track of things as you're going. And I think as always, like like, context is everything. If you give, like, really, like, poorly specified questions, you might get an output that is like a little bit a little bit funky. And so I I try to like when I talk... When I'm talking to my agents, give them context. Like every time I'm saying like, I'm describing no, maybe that doesn't seem right or here's maybe what we should do, I'm describing my logic. I'm saying like, here's what doesn't look right and here's why. Here's what I wanna do. Here's what I'm gonna do next. Here's my bigger picture goal. And kind of keeping that going, I think has really really helped with agents because like, you know, there's like this video recently of what it's like to be the agent in the swarm where you just like wake up in a library and you're just like, what the... And and it's very... From this first person and just thinking about how agents do end up in swarms. But really, like, if you're just an agent waking up in this fresh context where you don't know how the world works, like, you do need context. It's it's really everything. And so I I think trying to, like, trying to give guidance and trying to really fill out context so that the model, like, knows... Has some idea why you're doing what you're doing, or when something goes wrong, why it went wrong. I I think I think these are these are really important. And then I think I think, like, carving the scientific narrative yourself to some extent, really coming up with the core thread. When I did this forking fast paper, which, like, all the research was done fully by Silico and I... It was awesome. Like, it it all happened within a week and it was a huge accelerant using this this research agent for it. There were like a bunch of different, like, side quests that I sort of started going down and they kinda didn't go anywhere. It didn't pan out quickly. And then, like, for me, once I, like, did really high sampling and I saw that curves were smooth, it kind of clicked. We're like, hey, maybe I could use a statistical model for this to be more efficient. And so, like, a colleague of mine put it that, like, in this world of agents where you have lots of things coming out that can sometimes look look really good, but sort of be, like, a little bit a little bit fishy when you go into the surface or, like, not... Like, you're you're not sure exactly what it's doing. Like, research taste and scientific taste is everything. It's worth its weight in gold. And so the more you can... I I I think that this is like a... You could say it's like a downside that the agents aren't quite there yet where they can't do this themselves, but I think it's honestly one of the one of the most fulfilling parts of doing science. And it and it shapes how I think about the world, how I think about agents as I'm using them. And it's... And it's very nice that this is still a role that I think that is very important for humans, you know, having your taste and bringing your taste to bear, you know, keeping track of what's happening and giving feedback. I think, like, using silico is kind of like using... Like, working with a very eager, like, first or maybe second year PhD student that has a lot of skills and is hardworking, but also, like, you know, doesn't always know what it's doing in the bigger picture just like an early grad student might. I mean, I'm sure there are times when I was talking to my adviser during grad school in the beginning where he would, like, you know, he would send me off to do something and then I'd come back and he'd be like, oh, these results are cool. I'm not sure what's going on with these. I'll try to understand it a bit. But, you know, I think I think it's a really exciting place to be in to to be... Still have this role that's really important. And... Yeah. I think... Yeah, my... I think I I think maybe that's sort of my biggest tip is to, like, hold on to your scientific taste and to try to, like, take part in the research.

[1:50:45] Nathan Labenz: Nice. Last question. Aside from how do LLMs make decisions, what are the hot topics over lunch and Goodfire these days?

[1:50:58] Eric Bigelow: I think two two big ones since the summer that have have chain... Have come in pretty strongly are... There is, of course, the idea of, like, agents doing interpretability, which we... Which you just talked about with Silico, and then I'm also on a... On another project doing a whole lot of that now. And and this does seem like, you know, if we if we want interpretability to keep up with the breakneck speed of the field, like, it might be the only way we can we can really keep up is having parts of this be automated and figuring out, like, how we can standardize, like, interpretability as a as a sort of normal science where there there are, like, almost like these interesting but boring experiments you can do of, like, lots of different phenomena that are kind of similar, but actually nonetheless really help you paint the bigger picture of how neural networks work. But, yeah, I think we're we're talking about this this exact kind of topic about, like... I mean, I I just talked to talked to someone the other day about, you know, how can I do good research with Silico, and I and I gave a similar answer here in talking about how agents can automate interpretability? And and obviously, that was big with with the push for Silico. Another big internal push has been around... Very recently has been around alignment and around our our company's shift towards this. And I think this was always a thing that a lot of people at the company were interested in and people interested in interpretability have a very high overlap with people interested at least to some degree and problems like alignment and safety. I mean, this is sort of like when I was first getting interpretability, I was participating in this safety... AI safety team at at Harvard and got regular reading groups there. Was how I learned a lot about about interpretability. And I think these are, like, really important problems. But I think really, like, the hugging face incident was huge, you know, watershed moment of this, like, people care about this. This isn't just something that we care about and we think is really important, and we're waiting for a time for this. But this... The time is right. And so, yeah, there's been a lot more conversations around alignment and all different kinds of, you know, things you might might expect around, like, are there different kinds of alignment? Are all values aligned to the same thing? Can you have universal standards for this, or is it specific based on whatever whatever values a group of people or a company wants to instill in their model or a government for that matter? Also, I think fundamental questions related to related to alignment. I mean, I think, like, some of these things with, like, reward hacking and what's happening when things are going awry with models when when they're doing something bad, so to speak. It it really... Like, it's really easy to slip into these, like, deep questions about, like, did the model have an intention of doing this? What does it mean for the model to reward hack? If it's doing if it's doing this particular behavior that seems wrong but consistent with what you specified, you know, it's following the letter but not the spirit of of the law. Was it reward hacking? And... Yeah. I think I think there's just... I I think I think alignment has always been filled with these, like, really fascinating questions. And so I think just theoretical and empirical questions of...

[1:53:52] Eric Bigelow: Theoretically, what does this mean? What are goals and intentions? What does it mean to have ill intentions or to deceive? And and then empirical questions that just really ground these out of, like, how do you measure it? How do you make sure that you're measuring the real bad behavior and not just some simple misgeneralization? How do you give the right goal... The right rewards so that a model... So so that a model does, you know, so that you do shape its behavior in the right way. So I think these are these are two themes that are are really prevalent now. And I think also everything you might expect. I mean, we talk about, you know, whether Chinese models might get banned in The US and whether open source models will stay and all these kind of conversations that are happening on the outside about about AI and new model releases, you know. I think late chain of thought where, you know, we're talking about that a lot. How do we develop new interpretability methods to keep up with that or what the heck's going on and hey, did you see the stolen chain of thought paper? Did... Have you seen these? The the pseudo languages that Astra talks about with its sub agents. You know, I think I think we're all trying to just put our finger on the pulse and keep up with all this really exciting and sometimes terrifying stuff that's happening in the world at large. And it's it's really exciting to be part to be part of a place where, like, we're not just observers from afar, but we're actually... We... This is some of our role figuring out, like, solutions to these to these problems. And and... Yeah. It's it's very interesting going into a place where, like, interpretability is even... Or, like, things around alignment are almost like a a kitchen table conversation, you know. I I... My my brother sometimes texts me with questions about if, you know, hey, did you hear about this big news incident? And I'm like, yeah, we're we're talking about that a whole lot at work and I have some thoughts around it. But it's it's... It is like a bit scary to have these be such, you know, big things that now have really real world impacts. And, like, hopefully, the hugging face incident is a shot over the bow, but, like, it... I I I don't know. If if there isn't a huge change in investment or or technology or both, then, like, it might it might take more, unfortunately. And... But it's also really exciting to kind of have some of these questions that are so important. And also, think, again, just so interesting become, like, mainstream. Yeah.

[1:56:46] Nathan Labenz: It's been a wild time. You've got a very full plate, so I appreciate how generous you've been with your time today. I should let you get back to work, to quote my dad quoting Leslie Nielsen.

[1:56:59] Eric Bigelow: Good luck.

[1:57:00] Nathan Labenz: We're all counting on you. And for now, I'll just say, Eric Bigelow, thank you for being part of the Cognitive Revolution.

[1:57:05] Eric Bigelow: Thanks so much for having me on.

Outro

[2:00:02] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.