The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering

Hello, and welcome back to The Cognitive Revolution. Today, my guest is Curt Tigges, member of technical staff at Goodfire, and with Nick Cammarata, co-author of the new paper, "Neurons as an Interpretable Basis," the more academic and more LLM-focused companion to Nick's recent viral essay, "Neural Networks Are Already Clean." This work directly challenges what had been the received wisdom, that neural network internals are such a tangled mess and that individual neurons are so frequently reused across disparate concepts that we can't realistically hope to decode them directly and instead need helper models, such as sparse autoencoders, to make sense of their internal states. Curt's position, as you'll hear, is that individual neurons are in fact much more directly interpretable than previously believed, that most neurons, in fact, have one main function, that these functions are often better thought of as verbs and control flow gates than as entity recognizers, and critically, that LLM agents are now capable of grinding through a network and describing each neuron's role with remarkable accuracy. We begin with a detailed overview of the LLM they study, Qwen 3.6-35B-A3B, a 35-billion parameter model containing 5 million MLP neurons, which are spread across 40 layers, each of which has 256 experts containing just 512 neurons each. At every layer, 8 experts are active in any single forward pass. We linger on the highly expressive architecture of modern SwiGLU MLPs, which provide 2 distinct sets of signals to each neuron, one linear and one gated by the SiLU activation function. This specific MLP architecture was proposed as a transformer improvement by Noam Shazeer in 2020, in a five-page paper where he famously wrote that, quote, "We offer no explanation as to why these architectures seem to work. We attribute their success, as all else, to divine benevolence." Six years later, I'm struck by how similar this structure is to some of the gated architectures we've seen in Mamba and other highly expressive state space models. And my own intuition is that these parallel paths allow the model to simultaneously gather semantically relevant information and effectively filter it for relevance. Interestingly, because the activation function does allow for negative values to be transmitted, each neuron can have different meanings when activated in its positive and negative directions. From there, we move on to the techniques that Curt used to support the idea that a model's large activations are all you need for solid performance. The key concept is sparsity. In short, Curt shows that you can ablate roughly 90% of the individual neurons, keeping only those with the strongest activations with surprisingly minimal performance degradation. In some cases, such as on the ARC-AGI puzzles, such intensive ablation actually improved performance. And while it did cause the model to struggle on reasoning tasks, the authors do show that a very small amount of additional post-training on the sparsely activating model is enough to recover a large majority of the performance loss. Having established that the model's weaker activations matter less than most had believed, we move on to discuss the Neuron Scientist, the LLM agent that they use to scale the work of characterizing individual neurons. Given a corpus of text and relevant interpretability tools, a frontier model generates and then iteratively refines paragraph-length hypotheses that attempt to describe what each neuron does. These hypotheses in turn are validated by asking another model to indicate based on the description where in a given bit of text that specific neuron should be expected to fire, and then showing that the original model's behavior is substantially restored relative to ablating the neuron entirely when the other LLM's guesses are patched in to the model's activations. Along the way, we also touch on monitoring as the first and most obvious application of this work, the small circuits called neuron communities they identify and begin to explore, the fact that individual experts within the mixture of experts architecture are also more interpretable than previously discussed, how this work relates to the current consensus understanding of how models store and recall facts, as well as the roles of the residual stream and attention mechanisms, the possibility that small activations may matter much more in long context scenarios, the path from here to ambitious reverse engineering, how we should be thinking about scaling laws for interpretability, and what, if anything, this work suggests we might ought to do differently in the training phase in order to produce more readily interpretable models in the future. The bottom line for me is that this work does constitute a significant step forward in interpretability. Where much of the field previously assumed that helper models were required to disentangle model internals, there is now good reason to believe that we can study neurons directly in a highly scalable way. At the same time, whether such low level understanding can be effectively laddered up to higher order understanding of circuits and behaviors remains to be seen. There is much more work to be done, and while I totally understand how hungry people are for hope after a summer of so much bad news about our prospects for alignment through behavioral training alone, we should be very careful, as Curt is throughout this conversation, not to jump to unwarranted conclusions. Interpretability remains unsolved. With that, I hope you enjoyed this deep dive into the surprising legibility of individual neurons with Curt Tigges, member of technical staff at Goodfire.

Watch now!

Thank you for being part of The Cognitive Revolution,
Nathan Labenz