AGI and Institutional Disruption: Sam Hammond in Conversation with FLI's Gus Docker

Full Transcript

Transcript

Gus Docker: Hello, and welcome back to the Cognitive Revolution. As we reach the end of 2023, we're still just 11 months into making the Cognitive Revolution, but I wanted to take a moment to say a big thank you to everyone who has made this show possible. Starting with our sponsors, from our first presenting sponsor, Omneky, who agreed to support the show before hearing even a single episode, to the household names you've heard from throughout the year, including Oracle, Masterclass, Shopify, and NetSuite. We appreciate your support. I also want to say thank you to the entire Turpentine team, and in particular, our producers, Natalie Toren and Vivian Meng, our editor, Graham Besalu, and of course, our super connector and the mastermind behind the idea for a network of expert hosted podcasts, Erik Torenberg. We've made nearly 90 episodes of this show, and I always emphasize that AI content ages poorly. With that in mind, we've turned many episodes around in just a day or two. Now, this is not the norm in the podcasting business, and it does create a lot of odd hours for the team. But they've been here for it every step of the way, and I have been extremely privileged to be able to focus 100% of my time and energy on the content itself, knowing that they have my back on all the logistics. Of course, I also want to thank our many incredible guests. They are an amazingly diverse group, hailing from at least a dozen different countries around the world with all sorts of different academic and professional backgrounds, but all united by an appreciation for the transformative, dare I say revolutionary, potential of AI. Collectively, they have helped me develop a far deeper understanding of AI technology than I had coming into the year, and I hope the same is true for you. Finally, I want to thank all of you, the listeners, for making this project not just worthwhile, but a success well beyond my expectations. Coming into the year, I had never created original content before, and I had no platform or audience anywhere. And to be honest, I wasn't really sure if anyone would want to listen to the deeply in the weeds, hopefully, anti-polarizing content that I wanted to create. On that front, I'm glad to say that I have been consistently pleasantly surprised. Thousands of you have become regular listeners, and some of our most challenging episodes, including my recent two and a half hour monologue about state space models, have been among our very best received. We end the year as the twentieth ranked technology podcast according to Spotify, thirty-first according to Apple, and I take that as a sign that something is definitely working. Nevertheless, when a friend recently asked what metrics we use to track performance, I responded that for me, the best indicator is the number and the quality of people who reach out to share feedback with me directly. So with that in mind, if you have any topic or guest suggestions or just general thoughts about how we can make the show more valuable, we would love to hear them. We have a ton planned for the Cognitive Revolution in 2024, including episodes about the rise of AI automated scientific exploration, the intersection of AI and biology, the energy demands of AI, the future of robotics, and much, much more. So while we're working on all that, as a bit of bonus content, we're going to repost a few of my favorite AI scouting episodes from other shows, episodes that I was jealous of when I first heard them, starting today with an episode from the Future of Life Institute podcast in which host, Gus Docker, interviews economist, Samuel Hammond. This conversation is super wide ranging, covering the most likely default path to AGI, the economic and institutional transformations that AI will beget, the critical distinction between thinking about AI in isolation versus as part of a dynamic system, the proposal for a Manhattan or Apollo like mega project for AI safety, and lots more along the way. I hope you find this and the next couple of episodes to be helpful compliments to my own AI scouting work. And I encourage you to subscribe to the Future of Life Institute podcast feed, where you can hear lots more about AI and other issues of global importance. With that, happy New Year from the Cognitive Revolution, and here's Gus Docker and Samuel Hammond.

Gus Docker: Welcome to the Future of Life Institute podcast. My name is Gus Docker, and I'm here with Samuel Hammond, who is a senior economist at the Foundation for American Innovation. Samuel, welcome to the podcast.

Samuel Hammond: Hi, Gus. Thanks for having me.

Gus Docker: Fantastic. All right. I have so much I want to talk to you about, but I think a natural place to start here would be with your timelines to AGI. Why is it that you expect AGI to get here before most people?

Samuel Hammond: Well, I don't really know what most people think. I think the world divides into people who are paying attention and people who are basically normies. And in my day job, I work on Capitol Hill and in Washington, DC talking to folks about AI. And if you think about what people's implicit timelines are, you can read out people's implicit timelines by their behavior. I know Paul Christiano has short timelines because he's doubled up into the stock market. He's practicing what he preaches. But then when you have Sam Altman testifying to Congress, I like to say people are taking him seriously, but not literally. He's saying we're going to develop something like AGI potentially this decade and superintelligence thereafter. And then you have folks like Senator Marsha Blackburn being like, what will this mean for music royalties? And when the focus of policymakers is things like music royalties or the impact on copyright, it's not that those are invalid issues. It's that they belie relatively longer timelines. And then we also have this definitional confusion where folks like Yann LeCun would say AGI is probably decades away because he is using AGI to mean something that learns like a human learns in the sense that it's born as a relative blank slate and can acquire language with very few examples. So people have these moving goalposts of what they mean. For me, I think we can avoid those definitional conflicts if we just talk about human level intelligence. And humans are quite general. We're generally intelligent. That's what separates us from animals in a lot of respects. And when you look at how machine learning models are being trained today, like large language models and now multimodal models, they're being trained on human data. And they're being trained to reproduce the kinds of behaviors and tasks and outputs that humans output. And so they're an indirect way of emulating human intelligence. And so if you benchmark AI progress to that, then you can put information theoretic bounds on what's the likely timeline to basically an ideal human emulator, something that can extract the base representations, the internal representations of our brain through the indirect path of the data that our brain generates.

Gus Docker: Yeah. You have an interesting sentence where you write that AI can advance by emulating the generator of human generated data, which is simply the brain. Do you think this paradigm holds all the way to AGI?

Samuel Hammond: I think it holds this decade to systems that in principle can, in context, learn anything humans do. Again, this is a semantic question. Do you want to call that AGI or not? I think there are still outstanding issues around the limits of autoregressive models for autonomy and the question of real time learning. The way we train these models, we're freezing a crystal in place and humans are continuously learning. So there still are genuine potential architectural gaps, but from the practical point of view, from the economic point of view, we don't need to debate whether something is conscious or whether something learns strictly the way humans learn if it demonstrably can do the things humans do, right? And that goes to the original insight of the Turing test. It's sometimes presented as a thought experiment, but what Alan Turing was getting at was if you can't distinguish between the human and a computer, in some ways, indistinguishability implies competence. And we can broaden that from just language because arguably we've surpassed the Turing test, at least a weaker version of it, to human performance on tasks in general. If we have a system that can output a scientific manuscript that experts in the field can't distinguish from a human, then debating whether this is real AGI or not is, I feel, academic.

Gus Docker: It is surprising in a sense that when you interact with GPT-4, for example, and it can do all kinds of amazing things and organize information, present information to you, but then it can't or at least at some point, couldn't answer questions about the world after September 2021 or a date like that. That would be surprising if you presented that fact to an AI scientist 20 years ago. For how long do you think we'll remain in this paradigm of training a foundational model and then deploying that model?

Samuel Hammond: I mean, it's worse than that. I think it surprised people five years ago. Progress is moving along two tracks. There's the industry track and the pure research academic track, and they're obviously having feedback with one another. The pure industry track is just looking to create tools that have practical value and can improve products and so forth. And Meta has their own GPU cluster and they're training models so they can have fun chatbots in their messenger. And so those kinds of things are going to progress, I think, well within the current paradigm because we know the paradigm works, basically deep learning and transformers. And there's lots of minutiae on the side, but that basic framework seems to be quite effective and just scaling that up because we haven't hit the range of irreducible loss and transformers can do. Meanwhile, there's also this parallel pure research track where people on seemingly a weekly basis are finding better ways of specifying the loss function, ways of improving upon power loss scaling and all these different—sometimes they're new architectures, but often they're just bags of tricks. Those bags of tricks, to the extent that they comport with the paradigm industry's running with, they can be reintegrated and end up accelerating progress in industry as well.

Gus Docker: So do you think scale of compute is the main barrier to getting to human level AI?

Samuel Hammond: Yes. I mean, it's not all we need, but it's the main unlock.

Gus Docker: To what extent can more compute be used to trade off for lower quality data or for lower quality algorithms? Can you just throw more compute and solve the other factors in that equation?

Samuel Hammond: It depends on the thing you're trying to solve for. In principle, if we're talking about mapping inputs to outputs, then transformers are known to be universal function approximators. And so the answer is yes. That doesn't mean that they're necessarily efficient at approximating everything we want them to approximate. And sometimes universal function approximation theorems can be kind of trivial because they'll be like, okay, if your neural network has infinite width, then, yes, we can approximate everything. The key fact is both that they're universal approximators and also that they're relatively sample efficient, at least relative to things we found in the past. And so that to me suggests that, yes, they can compensate for things that they're bad at. On the other hand, the way research is trending is towards these mixed models, ensembles of different kinds of architectures, things like the recent Q-Transformer—and that's from Google DeepMind—uses a combination of transformers and Q-learning to have the associational memory and sample efficiency of transformers with the ability to assign policies to do tasks that you get from reinforcement learning. So, I imagine that there's going to be all kinds of mixing and matching. The key point is that in that space of architectures, it's a relatively finite search space. And as an economist, economists believe that supply is long run elastic. There's this famous bet between Paul Ehrlich and Julian Simon vis-à-vis the population bomb and whether population growth would lead to a Malthusian purge. And Julian Simon, being the economist, recognized that if prices rise for these core commodities, then that will spur research and development into extracting new resources, right? So he didn't have to know that fracking would be a technology. He understood that if oil prices went too high, people would find new oil reserves. And I think I have an analogous instinct when it comes to progress in deep learning, meaning you can become too anchored to the current state of the literature, but over a 10 year horizon, you can say, well, there's a huge gold rush to find the right way of blending these architectures. And I don't need to know in advance which is the right way to do that to have high confidence that someone will find it.

Gus Docker: Yeah, we can sometimes, if we're too deep in literature, we might lose the focus on the forest for the trees in a sense. And if we zoom out, we can just see that there's more investment, there's more talent pouring into AI and so we can predict that something is going to come of that. You have lots of interesting insights about information theory and how this can help us predict AI. What's the most important lessons from information theory?

Samuel Hammond: The reason I start there is because it's within the conceptual realm, it's the most general thing that bounds everything else. And when you look back at the record of, say, Ray Kurzweil, I first read The Age of Spiritual Machines when I was a kid, and in there, he makes a prediction that we'll have AIs that pass the Turing test by 2029 or so.

Gus Docker: And when was this book written?

Samuel Hammond: 1999.

Gus Docker: Yeah. That's pretty good.

Samuel Hammond: Right. And people will complain that he got things wrong because he said, well, I'll be wearing AR glasses by 2019, when in fact, Google Glass came out in 2013 and now we have Meta Glasses five years later. So he was wrong on the exact timing, but sort of right where the technology was, wrong where the minimal viable product was. But nonetheless, if you look at his track record, it's quite good for a methodology as relatively stupid as just staring at Moore's Law and extrapolating it out. And I think that reveals the power of these information theoretic methodologies to forecasting because they set bounds on what will be possible. The team at Epoch AI have a forecast called the direct approach where you can think of it as a way of putting bounds on when we'll have AIs that can emulate human performance through an information theoretic lens where they're looking at how much entropy does the brain process and how much compute will we have over time and what's implied by AI scaling laws. You put those three things together and you can set bounds on when we'll basically be able to brute force human level intelligence. And of course, that's an upper bound because we're going to do better than brute force. We're going to also have insights from cognitive science and neuroscience and also ways of distilling neural networks and so forth and better ways of curating data. So their modal estimate for human level AI is 2029 and their median is like 2036. And I talked to the authors and they lean towards the 2029, 2030 for their own personal forecasts. And so going back to, am I out on a limb here? I think among our circles, probably not. But among Congress and among the broader public, I think people are—they think everything's an asymptote. They're imagining, okay, we have these chatbots and they're not seeing the next step. I see a very smooth path from here to systems that can basically in context learn any arbitrary human task. And so what does that look like? It looks like systems that can basically sit over your shoulder or can monitor your desktop, your operating system as you work and watch you for an hour or two and then take over.

Gus Docker: And that'll be key to overcoming lack of training data, or why is it important that they can learn in context?

Samuel Hammond: Well, in-context learning is the secret sauce of the power of transformer models. They learn these inductive biases and induction heads and so forth that let them few-shot learn different tasks. So GPT-4 is very good at zero-shot learning on a variety of different things, it's incredibly good at few-shot learning. If you give it a few examples, it can kind of pick up where you left off. When I think about myself, when I want to learn a new recipe, I can go read a recipe book, but often what I prefer to do is to go on YouTube and watch someone make that recipe. And just by watching that person put together the stir fry, I have enough of a world model and enough knowledge of how to cook in general that I can in-context learn how to pick up from there and do that recipe myself. LLMs do that already. Multimodal models are increasingly doing that. Some of the recent progress in robotics, like I mentioned, the Q-Transformer paper, it shows that you can basically build robots with a basic world model and then have it learn new tasks with fewer than 100 examples of a human demonstration. So the human demonstrates the task and the robot can pick it up and take it from there. And why that's important is both for understanding the trajectory of AI but also its economic implementation because we're used to automation being this thing where you get a contract from IBM and you spend many millions of dollars with consultants and they build you some bespoke thing that doesn't really work very well and requires lots of maintenance. And so people have this prior that AI, even if it's near, will be rate limited by the real world because of all the complexity of implementation. But the point is if you have things that can in-context learn and perform as humans perform, then you don't need to change the process. You can take human designed processes and have the AI just fill in for the human. And so it leads to this paradox where we're probably going to have AGI before we get rid of the last fax machine.

Gus Docker: Yeah. When we think of, say, old IT systems in large institutions, we might think of moving from analog storage of information to the cloud. That's still going on in some institutions. That transformation has taken over a decade now. And so what exactly is it that makes AI different here? Is it that AI plugs in directly where the human worker would be?

Samuel Hammond: Yeah. Precisely. You don't need to redesign an existing process to plug into the automation and that applies both for the structure of tasks, right, so much of mechanical automation takes something like the artisanal work of a shoemaker and has to translate it into something repetitive that a machine or an automatic seamstress can do over and over and over again. Our older school kind of automation requires collapsing a task into a lower dimension so that simple automations can handle it. But when you have AGI, the whole point is generality. It's a flexible intelligence that can map to existing kinds of processes. So that's why I think this will catch people by surprise because it's not just that AGI could be this decade, but that when it arrives and crosses some thresholds of reliability, the implementation frictions could be very low.

Gus Docker: And do you expect AI would have to get all the way there in order to substitute for a human worker? I mean, I would expect it to be a bit more gradual than that, taking over, say 20% of tasks before 40% of tasks, 60% of tasks and so on. But here we're imagining that the AI kind of plugs in for the human worker for all tasks or what do you have in mind?

Samuel Hammond: These things are, yeah, you're right, much more continuous. It's not an on or off switch in part because the requisite threshold of reliability varies by the type of task. Arguably, self-driving cars like Waymo or Tesla have matched human performance, but regulators want them to be 100x better than human before they're loose on the road because of safety. Code completion and coding models are arguably still much worse today than elite programmers, but everyone is using them because even if it generates 50% of your code and you have to go back in and debug, it's still a huge productivity boost. So I think it will vary by occupation, by task category, modulo the risks and stakes involved in those tasks.

Gus Docker: Yeah, I guess then the question is how many of our jobs fall into the "it's more like self-driving cars" and how many of our jobs is more like programming?

Samuel Hammond: Right. I mean, I've been in a manager position before and I've had research assistants and interns and I know that they're a very lossy compression of the thing I want to do. And so they require oversight and co-piloting. We're in that stage now with AIs and a variety of different tasks. I recently read a paper evaluating the use of GPT-4 for peer review in science, and it found that GPT-4 would write reviews of work that bore some striking correlations with the points raised by human reviewers, but also left some things out. And so it concluded by saying GPT-4 could be an invaluable tool for scientific review, but it's not about to replace people. And that's just a case of, okay, give it five years.

Gus Docker: Yeah. This is a phenomenon you often see with some AI models out there and it has some capabilities but lacks other capabilities. And then people might kind of over anchor on the present capabilities and not foresee the way the development is going. I think people are continually surprised at the advancement of AI.

Samuel Hammond: Yeah. Absolutely. And Ramez Naam, the sci-fi author and futurist and energy investor, he gives this talk on solar energy and other renewables, and he has this famous graph where he shows the International Energy Agency, the IEA. Every year, they put out this projection of solar build out, and every year, it's like a flat line. But it's like a flat line on an exponential—the real curve is going vertical, and every year their projection is that it's just going to plateau. And I feel like people make that same mistake. And it sort of has this ironic lesson, to the extent that we're drawing parallels with the way our brain works and the way these models work, it seems like humans have a very strong autoregressive bias. So what's going on there?

Gus Docker: Is it an institutional problem, or is it a psychological problem? Why is it that we can't project correctly in many cases?

Samuel Hammond: To what you just said, I think it's probably both, but largely psychological. Our brains are evolved for hunter-gatherer societies that didn't really change over millennia. And even the last 40, 50 years have been a period of relative stagnation where we have a lot of pseudo innovation. And so I think people are just a bit disabused.

Gus Docker: Okay, you have some super interesting points about comparing the human brain, how the human brain works to how neural networks learn. What is universality in the context of brain learning and neural network learning?

Samuel Hammond: So universality is a term of art. It refers to the fact that different neural networks independently trained even on different data will often converge on very similar representations in their embedding space of that data. And you can extend that to striking parallels or isomorphisms between the representations that artificial neural networks learn and that our brain appears to learn. Probably the area of the brain that's been studied the most is the visual cortex, and it seems to me as a layperson that the broad consensus in neuroscience is that the visual cortex is very similar to a deep convolutional neural network—it's basically isomorphic to our artificial deep convolutional neural networks. And you train a CNN on image data and our brain is trained on our sensory data and it turns out they end up learning strikingly similar representations. And there are a few reasons for that. One is hierarchies of abstraction. It makes sense that early layers in a neural network will learn things like edges and simple shapes, and only later in the network, only deeper in the network will you learn more subtle features. So there's that sequencing part of it. And then there's also just the energy constraint. Gradient descent isn't costless. It requires energy. It requires a lot of energy. These data centers suck up a lot of energy. The same is true of our brain. Our brain consumes a lot of energy, like 25% of our calories, and especially when we're young, there's a very strong metabolic cost associated with neuroplasticity. Our brain being something shaped by evolution was obviously very energy conscious and so those energy constraints greatly shrink the landscape of possible representations from this infinite landscape of all the ways you could represent certain data to a much more manageable set of representations. And that doesn't guarantee that we'll converge on the same representations. It's at least suggestive of a weak universality where even when we don't have the exact same representations, they're often a coordinate transformation away from each other.

Gus Docker: It's actually a bit surprising. As you mentioned, when we train neural networks, we don't have the same energy constraints as the brain had during our evolution. And I would expect, again, from evolution, the human brains have many more inbuilt biases and heuristics. But if we then compare the representations in neural network to those in a human brain, we found that they are quite similar. Isn't that the whole point of universality? So does the neural network have the same heuristics and biases that we have or what's going on here?

Samuel Hammond: Well, one of the primary biases in stochastic gradient descent is sometimes called a simplicity preference, basically an inductive bias for more parsimonious representations. Parsimonious in the sense of Occam's razor, right? And that's a byproduct of this information theoretic concept of Kolmogorov complexity where Kolmogorov complexity is measured by—is there a short program that can reproduce this longer sequence? And if you can find a short program that's a more compact or more compressed way of representing it. And when you're under energy constraints, you're looking for those more compressed representations. And so that simplicity bias seems to be also the origin of generalization, of our ability to go beyond merely memorizing data, overfitting our parameters to finding a simpler way of representing those parameters, where we go from fitting a bunch of data points to recognizing, oh, these data points are being generated by a sine function. So I can replace all these data points by a simple circuit for that sine function or something like that.

Gus Docker: What can we learn about AI progress when we consider the hard steps that humans and our ancestors have gone through in evolution?Samuel Hammond: It's beyond evolution. This often comes up in the discussion of the Fermi Paradox. Life on Earth to exist at all, let alone intelligent life, had to pass through many hard steps. We had to have a planet in a habitable zone. We had to have the right mix of organic chemicals in the Earth's crust and so forth. We had to have the conditions for abiogenesis, the emergence of the very earliest sort of nonliving replicators, probably some kind of polymer type of crystal structure. Then we had to have the transition from single cell to multicellular organisms, the transition through the Cambrian explosion. Every one of these steps you could think of as a very unlikely, improbable thing, all the way up to the development of warm blooded mammals and social animals that were heavily selected for brain size to then the sociocultural hard steps of moving from small group primates to settled technological cultures than technological hard steps like the discovery of the printing press or the discovery of the transistor. You put those all together and life seems just incredibly unlikely. And this often goes to the point of view that creationists or intelligent designers would put forward. But then you zoom out and then you recognize, oh, wait. There are trillions of galaxies each with hundreds of billions of stars and hundreds of billions of trillions of planets. There's an awful lot of potential variation out there. And meanwhile, every one of these hard steps seems characterized by a search problem that is very hard. But then once you find the correct thing, like the earliest self replicator, things kind of take off.

Gus Docker: Hey, we'll continue our interview in a moment after a word from our sponsors.

Samuel Hammond: So you imagine that before the earliest self replicator, there were millions or billions of attempts to self replicate that didn't succeed?

Samuel Hammond: Yeah, it's just a huge search problem. And maybe there are more gradual intermediate stages where you have sort of—everything in biology ends up looking way more gradual the more you learn about it. But there are these phase transitions where you tip over and you get the Cambrian explosion or you get the printing press and the printing revolution. And so those hard steps end up looking relatively—they look more easy in retrospect because even though the search was hard, once you've tripped over the correct solution, there's sort of an autocatalytic self-reinforcing loop that pulls you into a new regime. And indeed, when you look at the emergence of life on Earth relative to the age of the universe, and Avi Loeb with some coauthors have done this, life on Earth is incredibly early. The universe is 13.7 billion years old, but life couldn't emerge really much sooner. The reason being the universe started out as hot and dense, had to cool down, stars had to form, those stars had to supernova so they could produce the heavy elements that are essential to life. And then those solar systems had to then take shape and then had to further cool so the solar system wasn't being irradiated constantly. And when you put all those factors together, human life emerged basically as soon as it was possible for life to emerge anywhere. And so this is one way to answer the Fermi Paradox is that we're just in the first cohort. But it also should give you strong priors that passing through those hard steps isn't as hard as it looks.

Gus Docker: And what's the lesson for AI here?

Samuel Hammond: Developing AGI is sort of a hard step. We're doing this kind of gradient search for the right algorithms for the right—what have you. And we seem to be now in a slow takeoff where we've figured out the core ingredients and there's now an autocatalytic process that's pulling us into a new phase.

Gus Docker: And what do you mean by autocatalytic?

Samuel Hammond: Self-reinforcing. Once it gets started, it sort of pulls itself. It sort of has an as-if teleology. You see this in nature, but you also see this in capitalism.

Gus Docker: And you would expect us to get to advanced AI basically as soon as it's computationally possible?

Samuel Hammond: It basically seemed that way. There was a kind of tacit collusion between Google and other players in the space to—they had transformer models since 2017, but really some of the precursors to transformers go back to the early nineties. But once you have this sort of profit opportunity that's in the background, it's hard in the competitive environment to stop an OpenAI from being like, oh, let's chase those profits. And then once that ball gets rolling, it's basically impossible to stop. This is why, whatever the merits of the pause letter, it's virtually impossible to really have a pause in AI development because everything is structured by these game theoretic incentives to just keep going faster. Once you've stumbled on the gold reserve, it's hard to just keep the prospectors from running there.

Gus Docker: Samuel, is the US government prepared for advanced AI?

Samuel Hammond: No. No. I mean, where do I start? I mean, the US government's in—if you think of it from a firmware level, many countries have national IDs. The US doesn't have a national ID. We have social security numbers. They're at least 9 digit numbers that date back to 1935. We have the core administrative laws date back to the early 40s. Much of our sort of technical infrastructure, the system the IRS runs on, date back to the Kennedy administration and are written in Assembly Code. There's also been this general decline in what you could call state capacity, the ability for US government to execute on things. And you hear about this all the time. You hear about how the Golden Gate Bridge was built in 4 years or something like that, and now it takes 10 years to build an access road. One of the reasons for that goes to what the legal scholar Nicholas Bagley has called the procedural fetish. Really, since the seventies, the machinery of the US government has shifted towards a reliance on explicit process. And proceduralism has pluses and minuses. If you have a clear process, government can kind of run on autopilot to an extent, but it also means you limit the room for discretion and you limit the flexibility of government to move quickly. And moreover, in our adversarial legal system, you also open up avenues for continuous judicial review and legal challenge, where famously New York has taken over 3 years to approve congestion pricing on one of their bridges because it has to undergo environmental review, and people who don't want to pay the congestion price keep suing.

Gus Docker: Do you think having more procedures would make it easier for AI to interface with the government?

Samuel Hammond: I would say having fewer procedures would make it easier for government to adapt.

Gus Docker: My assumption would be that having something written down, having a procedure for something would make it easier for AI to plug AI into that procedure. If it's less opaque and more kind of almost like an algorithm step by step?

Samuel Hammond: Yes. But the analogy I would give is to the Manhattan Project. The original Manhattan Project was run like a startup. You had Oppenheimer and General Leslie Groves sort of being the technical founder and the type A get things done founder. And they broke all the rules. They pushed as hard as they could. They're managing at its peak 100,000 people in secret, and they built the nuclear bomb in 3 years. And so the way we would do that today under procedural fetish framework would be to put out a bunch of requests for proposals and have some kind of competitive bid. And then we'd probably get the lowest cost bid, and it would be Lockheed Martin, and they would build half an Atom Bomb, and it would take 20 years and 5 times the budget. And so that's sort of what I'm getting at. It's not about process versus discretion per se. It's about the way process hobbles and straitjackets our ability to adapt and sort of represents a kind of sclerosis, a kind of sort of crystallized intelligence. We lay down the things that worked in the past as process and sort of freeze those processes in place ossifying a particular modality. And when the motor of production shifts and you need to completely tear up that process root and branch, it's very difficult because often there's no process for changing the process.

Gus Docker: Yep. I wonder if there are lessons for how government will respond to AI in thinking about how governments responded to, say, historical technical innovations of a similar magnitude, like the industrial revolution or the printing press or maybe the Internet computer. Do you think we can draw general lessons, or is it so specific that we can't really extract information about the future from them?

Samuel Hammond: I think there are very powerful general lessons. I think one of the first general lessons is that every major technological transformation in human history has preceded an institutional transformation. Whether it's the shift from nomadic to settled city states with the agricultural revolution or the rise of modern nation states or the end of feudalism with the printing press to in the New Deal era, the sort of transition with industrialization from the kind of laissez-faire classical liberal phase of eighteenth century America to an America with a robust welfare state and administrative bureaucracies and really an all new constitutional order. And so there's sort of better and worse ways for this transition to happen. There's sort of the internal regime change model, and you can think of Abraham Lincoln or FDR as inaugurating a new republic, a new American republic, or there's a scenario where we don't change because we're too crystallized and sort of like an innovator's dilemma get displaced by some new upstart. And different countries have different abilities and different sort of capacities for that internal adaptation. As a Canadian, I'm a big fan of Westminster style parliamentary systems. And one of the reasons is because it's very easy for parliamentary systems to shut down ministries, open up new ministries, to reorganize the civil service because it's sort of vertically integrated under the prime minister's office or what have you. In The US, it's much worse because given the separation of powers, Congress and the executive are often not working well together, just as an understatement. But then moreover, the different federal agencies have a life of their own. Often they're self funded and all these other things that make it very difficult to reform.

Gus Docker: Do you think Canada responded better to the rise of the internet than The US, for example? Isn't there something wrong with the story because The US kinda birthed the Internet and Canada adopted the Internet from The US?

Samuel Hammond: Let's compare, first of all, the impact of the Internet on weaker states because Canada and The US are similar or are sort of in one quadrant. They have differences, but the differences are small compared to other countries. If you think about internet safety discussions that would have been taking place in the early 2000s, people would have been talking about identity theft, credit card theft, child exploitation, these kind of direct first order potential harms from the internet. They didn't foresee that concurrent with the rise of mobile and social media that the internet would enable tools for mass mobilization simultaneous with a kind of legitimacy crisis where the sort of new transparency and information access that the Internet provided eroded trust in government and trust in other institutions. So you have these two forces interacting, the internet exposing government and exposing corruption and leading to a decline in trust while also creating a platform for people to rise up and mobilize against that corruption. And it's something that kind of rhymes with the printing press and the printing revolution where you had these sort of dormant, suppressed minority groups like the Puritans or the Presbyterians, the nonconformists. And with the collapse of the censorship printing licensing regime—they actually had a licensing regime in the UK Parliament back circa 1630. That licensing regime collapsed around 1634 or something around there. And that was 5 years before the English Civil War. And you see something like this in the Arab Spring, where the internet quite directly led to mass mobilization in Cairo and Tunisia and elsewhere and led to actual regime change, in some cases, through temporary state collapse. And that's because those were weaker states that hadn't democratized, that hadn't sort of had their own information revolution earlier in their history the way we did. In some ways, the American Republic is sort of the founding country built on the backbone of the printing revolution. So we were a little bit more robust to that because it's sort of part of our ethos to have this open disagreeable society. But clearly, the Internet has also affected the legitimacy of Western democracies. I think it's clearly one of the major inputs in sort of rising populism, the mass mobilizations that we see, whether in The US context, the 2020 racial awakening or the January 6th sort of peasant rebellion. These sort of look like the kind of color revolutions that we see abroad. And some people want to ascribe conspiracy theories to that. I think there's a simpler explanation, which is that people will self organize with the right tools. Our state hasn't collapsed yet, but clearly there are a lot of cracks in the foundation, if you will.

Gus Docker: Would it be fair to say that the main lesson for you from history is that technological change brings institutional change?

Samuel Hammond: Yeah. Not necessarily one for one. I'm not kind of a vulgar Marxist on this, but yes. And the reason for that is because institutions themselves exist due to a certain cost structure. And if you have general purpose technologies that dramatically change the nature of that cost structure, then institutional change will follow.

Gus Docker: Yeah. And I think we want to get to that. But before we do, I think we should discuss AI's impact on the broader economy. So not just the government, but the economy in general. Economists have this fallacy they point out often, the lump of labor fallacy. Maybe you could explain that.

Samuel Hammond: The lump of labor fallacy is essentially the idea that there's a fixed amount of work to be done. If you were thinking about the Industrial Revolution and what would happen to the 50% of people who are in agriculture, you couldn't imagine the new jobs that would be created, but new jobs were created. And the reason is because human wants are infinite and so demand will always fill supply. The second reason is because there's a kind of circular flow in the economy where one person's cost is another person's income. Society would collapse if we had true technological unemployment because there'd be things being produced but no one to pay for them. And so that ends up kind of bootstrapping new industries and new sources of production. There's still this open question is, is this time different?

Gus Docker: Yeah. That is that's exactly what I want to know. Because, I mean, for me, it's in retrospect, let's say, it's easy to see how workers could move from fields to factories, into offices. But if we have truly general AI, it's difficult for me to see where workers would move, especially if we have also functional robots and perhaps AIs that are better at taking care of people than other people are. I'm not asking you to predict specific jobs, but I'm asking you whether whether you think this historical trend will hold with the advent of advanced AI.

Samuel Hammond: The first thing to say is, when Keynes wrote Economic Possibilities for Our Grandchildren, a famous text where he predicted that technological progress would lead to the growth of a leisure society—

Gus Docker: And this was in the 1930s?

Samuel Hammond: Yeah. People have dismissed him as being wrong, but actually, you look at time use data and employment data, and people are working less. It didn't match the optimism of his projection, because it turns out maybe if we fixed living standards at what he expected, people want more and people will work more for more. But overall, people are working less. People do have more leisure. We've sort of moved to a de facto 4 day workweek. So there's one world where rapid technological progress sort of continues that trend and we all work less. It's sort of a technological unemployment that's spread across people and is enabled in part because in a world of AGI, maybe you only have to work a few hours a day to make $100,000 a year. There's another possibility which is that, well, AGI could in principle be a perfect emulation of humans on specific tasks. It can't emulate the historical formation of that person. So what I mean by that is if you had a perfect atom by atom replication of the Mona Lisa, it wouldn't sell at auction. Because people aren't just buying the physical substrate, they're also buying the kind of world line of that thing. And that's clearly the case in humans as well. There are certain talking heads that I go and enjoy not because they are the smartest or what have you because I'm interested in what that person thinks on this because they have a particular personality, a particular world line. And then the third factor is sort of artificial scarcity. And so even in a world with abundance and supply in services and goods, there are still things that will be intrinsically scarce, real estate being probably the canonical thing, but also energy and commodities and so forth. And the reason real estate is intrinsically scarce is because people want to live near other people and people want to live in particular areas of a city. They want to live in the posh part of town. And those are positional goods. We can't all live in the trendy loft. So that builds in the kind of artificial scarcity. And so people will still be competing over those things. This is sort of related to artificial scarcity, but there's also—I'll break it out into a fourth possibility, which are sort of tournaments and things that are structured as tournaments. Having chess bots that are strictly better than humans at chess hasn't killed people playing chess. If anything, more people play chess today than at any point in human history.

Gus Docker: Yeah. It's more popular than ever.

Samuel Hammond: Yeah. And the reason is because people like to watch other humans playing, and also they're structured as sort of zero sum tournaments where there can only be the best human. You look at other things that have been created just in the last 15, 20 years, like the X Games. I think people will still want to watch other people do the Olympics or do motocross and all these other things. And so maybe more of our life shifts into both maybe greater leisure on the one hand, more competition over positional goods, and more production that is structured as a tournament.

Gus Docker: Yep. Yeah. I can see many of those points. I'm just thinking, again, with fully general AI, you would be able to generate a much more interesting person playing chess or at least a simulation of a very charismatic and interesting human chess player. Why wouldn't people watch that chess player as opposed to to the most to the best human? Maybe they will.

Samuel Hammond: It's hard to know. The question is who's producing that video stream because you still need the human behind it that had the idea.

Gus Docker: You could imagine people being dishonest about the history of of this chess player. That this simulated chess player could be fully digital, fully fictional, so to speak, and just pretending to be human.

Samuel Hammond: Right. So it could fool people. That's the case too. No. I can't rule that out, but I would just say that however that person is monetizing their deepfake chess player, they're making money, which they're then spending back into the economy, and so they'll produce jobs somewhere.

Gus Docker: You think more people will move into, say, people focused industries like nursing and teaching? Is that a possible way for us to maintain jobs?

Samuel Hammond: Maybe nursing, at least in the short run. I'm not very long on education being being labor intensive for much longer.

Gus Docker: But you don't think education is at least, say, grade school education, is that really about teaching people or conveying knowledge? To what extent is it about conveying knowledge and to what extent is it about the social interaction and specializing your teaching to the individual student?

Samuel Hammond: Well, AIs are very good at customization and sort of mastery tutoring. Education is a bundle of things. And for younger ages, it's also daycare. It is socialization, like you said. At the very least, it suggests a reorganization of the division of labor because the types of teachers that you would select and hire for may differ if the education component of that bundle is being done by AI. Maybe you select for people who don't have any subject matter expertise but are just highly conscientious and good around kids. Or maybe you unbundle from public education altogether and it rebundles around a jujitsu school or a chess academy because you'll have the AI tutor that will teach you math, but you'll still want to grapple with a human.

Gus Docker: Yeah. Yeah. What about industries with occupational licensing like law or medicine? Will they be able to keep up their quite high wages in the face of AI being able to be a pretty good doctor and a pretty good lawyer?

Samuel Hammond: It's easy to solve for the far for long term equilibrium. With the rise of the Internet, you can do a comparison of the wage distribution for lawyers pre and post Internet. And circa the early nineties, lawyer incomes were normally distributed around $60,000 a year. After in the 2000s, they become bimodal. And so you have one mode that's still around that $60,000 range. Those are the family lawyers. And then you have this other mode that's into the 6 figures, and those are big law. It's the emergence of these law firms where you have a few partners on top and maybe hundreds of associates who are doing kind of grunt work using Westlaw and LexisNexis and these other legal search engines to accelerate drafting and legal analysis. So if that pattern repeats, I could imagine these various high skill knowledge sectors to also become bimodal, where in the short run AI serves as a copilot, sort of like Westlaw or LexisNexis was for legal research and enables the kind of 100x lawyer. So there's a kind of averages over dynamic. Longer run, you start to see the possibility of doing an end run around existing accreditation and licensing monopolies where, obviously, the American Medical Association and medical boards will be highly resistant to an AI doctor. I tend to think that they'll probably end up self cannibalizing because the value prop is so great even for doctors to do simple things like automate insurance paperwork and stuff like that. But to the extent that there is a resistance, to the extent that in 10 years there's still a requirement that you must have the doctor prescribe the treatment or refer you to a specialist even though the AI is doing all the work and they're just sort of like the elevator person that's actually just pushing the button for you. It'll be very easy to end run that because AI is both transforming the task itself but also transforming its the means of distribution. And if you can go to GPT-4 and ask for—put in your blood work and get a diagnosis—but no regulator is going to stop that. And so GPT-4o becomes sort of the ultimate doctor of headquarters.

Gus Docker: You write a lot about transaction costs and how changes in transaction costs change institutional structures. First of all, what are transaction costs and how do you think they'll be affected by AI?

Samuel Hammond: So transaction cost is sort of an umbrella term for different kinds of costs associated with market exchange. And this goes back to Ronald Coase's famous paper on the theory of the firm where he asked the question, why do we have corporations in the first place? If free markets are so great, why don't we just go and spot contract for everything? And the answer is, well, market exchange itself has a cost. There's the cost of monitoring. If you hire a contractor, you don't know exactly what they're doing. There's the cost of bargaining. Having to haggle with a taxi cab driver is a friction. And there's the cost associated with searching information. So taking those 3 things together, they're not all that companies do but they structure the boundary of the corporation. They explain why some things are done in house and some things are done through contracts. If there's high monitoring costs, you want to pull that part of the production into the company so that you can monitor and manage the people doing the production.

Gus Docker: And some of the same effects go for the existence of governments. Right?

Samuel Hammond: Yes. Because governments—with a certain gestalt, governments and corporations aren't that different. They're kinds of institutional structures that pull certain things in house and certain things are left for contracting or outsourced. And you can see sort of different kinds of governments having different parallels with different kinds of corporate governance. Relatively egalitarian democratic societies like Denmark are kind of like mutual insurers, whereas more hierarchical authoritarian countries are more like—Singapore, say, is more of a joint stock corporation. And indeed, Singapore was founded as an entrepot for the East India Company. So there are very deep parallels. And it's also essential—transaction costs are essential ones to understand why governments do certain things and not other things. All Western developed governments guarantee some amount of basic health care. But most outside of say the National Health Service in The UK, most of these countries guarantee the insurance. They don't necessarily nationalize the actual providers. And the reason goes to transaction costs and sort of an analysis of the market failure in insurance. Likewise with roads. It's possible to build roads through purely private means and indeed, countries like Sweden, a lot of the roads are run by private associations. But if you have lots of different boundaries, different micro jurisdictions and so forth, there can be huge transaction costs to negotiating up to an interstate highway system. And those transaction costs then necessitate public infrastructure projects.

Gus Docker: So the transaction costs in this case would be—as a private road provider, you'd have to go negotiate with 500 different landowners about building a highway, whereas the government can do some expropriation and simply build a road much faster or with less transaction costs, at least.

Samuel Hammond: Yeah, precisely. And we're seeing this dynamic in The US with permitting for grid infrastructure and transmission. We're building all the solar and renewable energy, to build the actual transmission infrastructure to get the electrons from where it's sunny to where it's cold requires building high voltage lines across state lines, across different grid regions, and there are all kinds of NIMBYs and negotiation costs involved, holdouts, and so forth. And so the more those kind of costs exist, the more it militates towards a kind of larger scale intervention that federalizes that process.

Gus Docker: Yeah. The big question then is how will AI change these transaction costs? What will the effects be here? Samuel Hammond: It's easy to say that they will be affected. Obviously, the Internet affected them to an extent. And we talked about the ease of mobilizing protest movements or the sunlight that was put on government corruption. Those are reflecting declines in the costs associated with information and coordination. I think AI takes us to another level and it's important to think through in part because right now the AI safety debate, at least in the United States, is very polarized between people who are like, everything's going to be great and people who are like, this is a Terminator scenario or an AI kill us all, existential risk. Even if we accept the existential risk framing, there's still going to be many intermediate stages of AI before we flip on the superintelligence. And those intermediate stages have enormous implications for the structure of the very institutions that we'll need to respond to superintelligence or what have you. The ways we can see this is because all these information and monitoring and bargaining costs are directly implicated by commoditized intelligence. You start with the principal-agent problem. There is no principal-agent problem if your agent does exactly as you ask and works 24/7, doesn't steal from the till. And so AI agents dramatically collapse agency cost. Monitoring. Now that we have multimodal models, in principle, we could have cameras in every house that are just being prompted to say, is someone committing a crime right now? Whether we wanted to go that direction or not, it gives you a sense of how the cost of monitoring have basically plummeted over the last two years and are going to go way lower. And so you're starting to see this roll out in the private sector with Activision has announced that they're going to be using language models for moderating voice chat in Call of Duty. And this is a more robust form of monitoring because in the past, you would have to ban certain words, like certain swear words or things associated with sexual violence. But then people could always get around those by using euphemisms. On YouTube, the algorithm will ding you if you talk about coronavirus or if you talk about murder or suicide, these things that throw off flags. What people have taken to doing is saying they were unalived rather than murdered. And that doesn't fool a language model. If you ask a language model, if you prompt it in a way to look for broad semantic categories, not just a narrow word, it's much more robust. And so what that means, with the you already start to see it, like I said, with Activision and the use of LLMs in content moderation, you're going to start you're going to see it in the use of multimodal models for productivity management and tracking. Microsoft is unveiling their 365 Copilot where you're going to have GPT-4 in Word and Excel and Teams and Outlook. But at the same time, you're also going to have a manager who is going to be able to say, to prompt the model, tell me who is the most productive this week. Something as vague as that. And so you see this diffusion in the private sector. The question is, does it diffuse in the public sector? There's obvious ways that it would be a huge boon. Inspector General GPT could tell you exactly how the civil service is working, whether there's corruption, whether there's a deep state conspiracy or something like that. And at first blush, a lot of what government does is kind of a fleshy API. Bureaucracies are nodes that apply a degree of context between printing out a PDF and scanning it back into the computer. It varies. There's degrees of human judgment that are required. But on first order, government bureaucracies seem incredibly exposed to this technology and in a way that could diffuse really rapidly because, going back to Microsoft 365 Copilot, Microsoft is the biggest IT vendor in the US government. And so you can imagine once everyone has this pre-installed on their computer that the person at the Bureau of Labor Statistics who's in charge of doing the monthly employment situation report, the jobs report, at some point he's going to be walking into work and hitting a button, asking Excel to find the five most interesting trends and generate charts, and the report is done. And in the private sector, that person will be reallocated and maybe doing things that the computer's not good at yet, but these positions are much stickier in government. To the extent that diffusion is inhibited on the public sector side, I worry about the kind of disruption and displacement of government services by a private sector that's adopting the technology really fast.

Gus Docker: This is something we'll talk about in a moment. Before that, I just want to get to your complaints about isolated thinking about AI. You've sketched out some complaint about people thinking about AI only applying to one domain and then not really seeing the bigger picture. What are some examples here? Why do you worry about isolated thinking?

Samuel Hammond: A few dimensions to this. One is what I've called the horseless carriage fallacy. The view that what automobiles were was just a carriage without a horse. And so that anchors you to the older paradigm and it's like you're changing one thing and everything else stays the same. And you neglect all the second order ways that the development of the automobile enabled the build out of highway systems, total reconfiguration of the economic geography, and then implications for institutions like the state where once you have road networks or telegraph networks or any of these kind of networks, it suddenly becomes easier to monitor agents of the state in other parts of the country and so you can build out more of a federal bureaucracy. And so all these things were second order and were kind of neglected if you just were too focused on the first order effects of displacing the horses.

Gus Docker: And in essence, the second order effects turned out to be much more consequential in the end.

Samuel Hammond: Yes. It always seems to be. And likewise with the Internet and how to think about AI use and misuse. There's lots of valid discussions there, but they're always very first order. And when you think about the way the Internet has disrupted legacy institutions, yes, there's disinformation. But often, the thing that's disrupting is not fake news. It's real news that's being repeated with misleading frequency. That's throwing off our availability heuristic. Or it's valid things, valid complaints, the protests in Iran. The protests in Iran had this striking parallel to the protests following the George Floyd protests and protests in other countries where they even have a three word chant. Or the case of the Arab Spring in Tunisia that started with the person self-immolating. There's sort of the structure that repeats where you have a murder or some shocking event. And because of the way social media is organized, it synchronizes people around the event in a way that's kind of stochastic. Like, it's like lightning striking. You don't know what event it's going to strike on. But once we're synchronized, then we start moving back and forth in a way that causes the bridge to buckle. Nothing about that is a misuse. Those are all valid uses, but their use results in collective action. It's solving not just for the partial equilibrium but the general equilibrium when everyone is doing this. And I think the person who wrote the best on this sort of conceptually was Thomas Schelling. And one of his little books, Micro Motives, Macro Behavior, had been influential on me as a kid where he talks about all these sort of toy models where you're at a hockey game or a basketball game, and something is happening, something exciting is happening in the arena. And so the people in front of you stand up to get a better view. And then you have to stand up to get a better view over them and so on. And so it cascades and suddenly everyone went from sitting to standing and no one's view has improved. And so these general equilibria where you solve for everyone's micro incentives and the kind of new Nash equilibrium that emerges, that ends up being the thing that drives the multiple equilibrium shift from one regime to another. And throughout, there may be no actual examples of misuse involved. It may just be people following their individual incentives.

Gus Docker: I think it's worth stressing this point you make about the effects of earlier AI systems on our institutions. They might have effects that deteriorate our institutions such that we can't handle later and more advanced AI. And ignoring this would be an example of isolated thinking and ignoring the second order effects, right?

Samuel Hammond: Yeah. And it also changes the agenda, right? The AI safety agenda shouldn't just be about the first order things or alignment, very important, but it's led to a discussion of do we need a new federal agency, and if so, what kind of agency? Whereas it may be more appropriate to think not what new agency do we need, but how do all the agencies change? And how do we sort of brace for impact and enable a degree of coevolution rather than displacement.

Gus Docker: I don't know whether the question of how to get our institutions to respond appropriately is more difficult or less difficult than the problem of aligning AI, but it certainly seems very difficult to me. So are we making it harder on ourselves if we focus on the second order effects on institutions?

Samuel Hammond: I mean, it's unavoidable. We can't pick and choose what kind of problems. But the alignment problem, the hard version, is yet to be solved. But we have many examples of governments building state capacity and having kind of shifting from very clientelistic, sticky, corrupt governments to sort of modernized governments where state capacity is built, then that government can sort of break out of the middle income trap and become rich.

Gus Docker: You mentioned Estonia as an example of a country that's pretty advanced on the IT front, on the technology side. Maybe you could talk a bit about Estonia.

Samuel Hammond: Yeah, I would just say in general, it's hard for any organization to reform itself from within when there is path dependency, but I would say at least we have examples of it being done where we don't have examples of alignment being solved yet. When it comes to Estonia, Estonia is an interesting case. It's sort of an exceptional case because after the fall of the Soviet Union and the breakup of the peripheral former Soviet states, they kind of had a blank slate. They also had a very young population and people who had a kind of hacker ethic within their civil service. And so with that blank slate and with that hacker ethic, they were very early to adopt and to foresee the way the internet was going to shape government through a variety of e-government reforms. So early in the late 90s and into the 2000s, were some of the earliest to digitize their banking system, like e-banking, to build this system called X-Road, which is kind of like a cryptographically secured data exchange layer. It resembles a blockchain, but it was about a decade before blockchain was invented.

Gus Docker: For exchanging information between different government entities. Your medical information could be uploaded to the system and then be available to all systems that have the right to see that information.

Samuel Hammond: Exactly. In a way that's cryptographically secured and distributed. So if a missile hit the Department of Education, you don't lose your education records because it's distributed. And that also enabled an enormous amount of automation where, for instance, this is my understanding, a child born in Estonia, once you file that birth record, it more or less initiates a clock in the system that will then enroll your child in school when they turn four or five, automatically because it knows that your child has aged and then unless it had a death record to cancel that out. That also means you can do taxes and transfers much simpler. You get your benefit within a week. You can integrate across different parts of public infrastructure, use the same card to ride the bus as you do to launch a new business. And it also serves as a kind of platform for the private sector to do government by API, to build new services on top of government as a platform and integrate with government databases.

Gus Docker: Yeah, and so the point here for us is that institutional reform is possible, modernizing government is possible, at least under certain circumstances. We have proofs of concepts of this happening.

Samuel Hammond: The hard thing is the path dependency. There's always a strong instinct to want to start from scratch, and it's normally not advisable because it's just too hard. And so this is why it's hard in the US. This is why you have African countries that leapfrog us in payment systems and so forth. The challenge of this decade or century is how do we solve that path dependency problem and how do we get to Estonia? It used to be get to Denmark. Now let's get to Estonia and find that pathway up mount probable.

Gus Docker: Great. Let's get to your wonderful series of blog posts on AI and Leviathan. In this context, what do we mean by Leviathan?

Samuel Hammond: Well, this all interrelates. So, Leviathan was the book Thomas Hobbes wrote at the start of the interregnum, after the end of the civil war. And it was basically his early political science, early defense of absolutist monarchy as a way to restore peace and order after a decade of infighting. And Hobbes kind of hit on some basic structural game theoretic properties of why we have governments at all. He talked about life being nasty, brutish, and short in the state of nature, war of all against all. And peace is only restored when people when people who don't trust each other offload enforcement and policing responsibilities to a higher power that can then restore a degree of peace and order. AI and Leviathan is talking about how does AI change the story? Does it reinforce the Leviathan? Does it lead to a digital police state, a la China, or is it something that we impose on ourselves? And we talked about how multimodal models could in principle be used to put a camera in everyone's house and have it just continuously monitoring for people doing any kind of crime. That's something that North Korea might do. In the US context, it's something that we're very liable to just voluntarily do to ourselves because we want to have Ring cameras and Alexa assistants and so forth. And so that leads to a kind of bottom-up Leviathan that is potentially no less oppressive and maybe even more oppressive because there's no one that we can appeal to to change the rules.

Gus Docker: Yeah. So Leviathan is one way to respond to technological change, but you mentioned two other ways we could alternatively respond.

Samuel Hammond: Right. So basically, anytime a technology greatly empowers the individual, it creates a potential negative externality. Hobbes called these our natural liberties. In the state of nature, I have a natural liberty to kill you or to strong-arm you. And governments exist to revoke those natural liberties, but for a higher form of freedom. And so anytime a technology greatly increases human capabilities vis-a-vis other humans, the three canonical ways we can adjust are, ceding more authority to that higher power, the Leviathan option, and then the other two options are adaptation and mitigation and normative evolution. So the example I give is if suddenly we all had x-ray glasses and we could see through walls and see through clothing. One option, we have a draconian totalitarian crackdown that tries to seize all those x-ray glasses. Another option is we adjust normatively, culturally that our privacy norms wither away and we stop caring about nudity. And then the other option is adaptation mitigation where we put mesh into our walls and wear leaded shirts and pants.

Gus Docker: Yeah. I guess continuing that analogy a bit between the smart glasses and AI. You have this amazing write-up of ways in which AI can increase the informational resolution of the universe. So you give some examples that are I'm thinking specifically of AI at identifying people by gait, for example.

Samuel Hammond: So gait recognition is nothing new. China has had advanced forms of gait recognition for a while now. So even if you cover your face, it turns out we're constantly throwing off ambient information about ourselves, about everything. And the way you walk, the particular gait that you have is a unique identifier. Another example is galaxy surveys. From Hubble Telescope to now the JWST, there have been tons of astronomical surveys of distant galaxies and so forth. And all of a sudden, all that old data, it's like that same data set is now more useful because applying more modern deep learning techniques, we can extract entropy that was in that data set that we didn't have the tools to extract yet and discover that there are new galaxies or other phenomena that we missed.

Gus Docker: Another example you give is listening for keystrokes on a keyboard and extracting information about a password being typed in, for example, which is something that, of course, humans can't do, but we can do with AI models.

Samuel Hammond: Yeah. So that was a paper showing that you can reconstruct keystrokes from an audio recording, including a Zoom conversation. So I hope you haven't typed in your password because people in the future. And so this goes to the fact that it's sort of retroactive that, even if the technology wasn't diffused yet, any Zoom conversation, any recording where someone typed their password in the future will be like those galaxy surveys where someone will go backwards in time and turn up the information resolution of that data.

Gus Docker: Yeah. This is pure speculation, but I wonder if I mean, imagine anonymized people in interviews, say, 10 years ago, whether they will be able to stay anonymous or whether AI will be able to extract the data about their face or their voice that wasn't technically possible when the interview aired.

Samuel Hammond: Yeah, exactly. There are already systems for depixelating. You probably could do something similar for the voice modulation. And then also going back to this ambient information we're always shedding, identifiers in the way we write, the kind where we place a comma, the kinds of adverbs we like to use and so forth. People just dramatically underrate how much information we're shedding, in part because we're blind to it.

Gus Docker: Some people who are taking great efforts to stay anonymous online, people in the cryptography space, for example, will put their writings through Google Translate to French and then back to English to erase subtle clues that could identify them personally. Why is AI so much better at tasks like the ones we just mentioned compared to humans?

Samuel Hammond: Well, it goes back to what we were talking about with putting information theoretic bounds on AGI. When you minimize the loss function in a machine learning model, you're trying to minimize the cross-entropy loss. And cross-entropy is how many bits does it take to distinguish between two data streams. And if it takes a lot of bits to distinguish between the two, that means they're relatively indistinguishable. So this going again to the Turing test. Like, if we have Turing test where I can tell right away that the AI is different from the human, that suggests a high cross-entropy. But if I could talk to it for days and do all kinds of adversarial questioning, I might still be able to, in the end, tell the difference between the two, but we've minimized that cross-entropy loss. And so when you have any arbitrary data distribution that you're trying to predict, whether it's trying to predict galaxies in astronomical data or passwords from fingerprint data on a phone screen. All these things embed a kind of physical memory of the thing in question and can often be reconstructed through this kind of loss minimization where you have a system that asymptotically extracts the entropy that was latent in the data. And this can be done in a way that is often quite striking where we can, with stable diffusion, make fairly accurate predictions of what people are imagining in their mind using fMRI data. And fMRI data is like blood flow data in the brain. It's a very lossy representation of whatever is happening in the brain. But there's still enough latent entropy in there that we can kind of reverse engineer or decompress it into a picture.

Gus Docker: And this could turn into a form of lie detection?

Samuel Hammond: Yeah. I think it already basically has. If you have fMRI data or EEGs or other kinds of direct brain data, it's probably a lot easier, but we already have systems that are over 95% accurate at detecting deception from just visual video recordings.

Gus Docker: We can see how all of this information that we are continually shedding gives rise to the possibility of a Leviathan, either of the private or of the government's kind. I wonder what role do you see open sourcing AI models playing here? What are the trade-offs and risks in open sourcing AI?

Samuel Hammond: Among the people who are most bullish on open source, there's often a kind of libertarian ethic undergirding it. Regardless of whether that's a good idea or not, one of the things I'm trying to communicate to that group is to say that be careful what you wish for because of these kind of paradoxical Hobbesian dynamics. The fact that in America, you never know if someone has a gun or not. On the one hand, the Second Amendment enhances our freedom. On another hand, you don't get the sort of everyone's doors unlocked and people are like the police in England don't even have guns. There's a certain freedom that derives from us not all being heavily armed. And likewise, with open sourcing powerful AI capabilities, it empowers you as an individual. But in general equilibrium, once we all have those capabilities, the world could look much more oppressive either because we're all spying on each other all the time and we can all see through each other's walls or because there's a backlash and the introduction of Leviathan-type solutions to restrict our ability to spy on each other all the time. And my general sense is that we can only delay and we can't really prevent things from being open source over the long run because there's a sort of trickle down of compute requirements. But in the interim, there are definitely things that are valuable to open source. Having a 70 billion parameter language model is not a threat. In fact, I think it's probably useful for alignment research for something like that to be open source. But if you are a researcher and you've developed an emotional recognition model that can tell with 99% accuracy whether someone is lying or not lying and whether your girlfriend loves you or not. Like, these things or the ability to see through walls using like, I talk about the use of Wi-Fi displacement. There are people who have built pose recognition models using the displacement of the electromagnetic frequency of your Wi-Fi, and they can see through walls. Like, what's the rush to put that on Hugging Face and to make it as democratized as quickly as possible. I would say that if we value the adaptation and mitigation pathway as opposed to the Leviathan pathway, then there's a value in slow rolling some of these things.

Gus Docker: How do you think government power will be or relative government power will be affected by AI? So you write somewhere in this long series of blog posts that AI will cause a net weakening of governments relative to the private sector. Why is that?

Samuel Hammond: Yeah, specifically Western liberal governments under constitutional constraints. So if you imagine society being on this kind of knife edge, I talk about this in the context of Daron Acemoglu's book, The Narrow Corridor, where he describes liberal democracy as sort of being in this corridor between despotism on the one hand and anarchy on the other, and we can sort of stay in this saddle path where society and the state are kept in balance. If you veer off that path, on the one hand, the state could become all powerful, and that's the sort of China model, more authoritarian digital surveillance state. And indeed, China built up their digital surveillance state and their Internet firewalls and so forth after watching the Arab Spring and seeing how the Internet was destabilizing to weaker governments. And so I fully expect that AI will be very empowering and self-reinforcing of the power of the Chinese government. Indeed, their draft regulations for large language models stipulate that you can't use the model to undermine national unity or challenge the government, and so they're baking that in. In liberal democracies, we think of ourselves as open societies, and the issue is that we're only open at the meta level. There's a public sphere. There's freedom of information laws. We have freedom of speech. I don't have freedom of speech if I walk into a Walmart. Walmart is private property. In open societies, it's not that we don't have social credit scores and forms of thicker forms of social regulation. It's just that we offload those functions onto private, competing private actors, whether it's a church that has very strict doctrines to be a member or other kinds of social clubs. The fact that these days, if you want to go to a comedy club, they'll often confiscate your phone at the door because they don't want you recording the comedian's set and putting it online. My anticipation is that because of those constitutional constraints that limit the ability of liberal democracies to go the China route, because of our civil laws or bills of rights and so forth, and also because of the law's procedural constraints. This will naturally shift into the private sector. And we see that already with the use of AI for monitoring and employment, for policing speech in ways that would be illegal if done by the state, but are fine if done by Facebook. To the extent that AI continues to increase these kind of negative externalities and therefore puts more value on having a vertically integrated experience, a walled garden that can strip out the negative forms of AI and reinstate a degree of harmony between people, more and more of our social life will be mediated through these private organizations rather than through a kind of open public sphere.

Gus Docker: You're imagining that government services will be gradually replaced by private services that are better able to respond, while governments fight to uphold individual rights. In Walmart or on Facebook, you are regulated in ways that the government couldn't regulate you, but you still have the choice to go to Target instead of Walmart or to X instead of Facebook. Isn't that the fundamental thing? So the fundamental thing is the choice between services and won't governments uphold citizens' rights to make those kinds of choices?Samuel Hammond: Yeah, no, I agree. And so this would be the defense of the liberal model is that we allow thicker forms of social regulation because it's moderated by choice and competition. And the issue with Chinese Confucian integralism isn't the fact that it's super oppressive. It's the fact that you only have one choice and you don't have voice or exit. So yeah, but it's obviously a matter of degree, right? When ride hailing first arose, I remember back in 2013, 2014, it wasn't that long ago. I think Uber was founded in 2009, but it really only started taking off in the early 2010s. People thought it was crazy to ride a car with a stranger. And then within five years, it was the dominant mode of ride hailing. And in that five year period, essentially, we saw a kind of regime change in micro where taxis went from being something that was regulated by the state through these commissions that were granted legal monopolies and used licensing and exams and other sort of brute force ways of ensuring quality to competing private platforms where you have Lyft or Uber to choose from, and they replace the explicit governance of legal mandates with the competing governance of reputation mechanisms, of dispute resolution systems, of structured marketplaces that collapse the bargaining frictions. Right? You never have to haggle with an Uber driver. You just sort of get in. And that was obviously a much better way of doing ride hailing. So even though there was sort of a violent resistance early on, literally, in France, they were throwing rocks off of bridges and cab drivers in New York were killing themselves. So for the people affected, it was a very dramatic sort of regime change, but for everyone else, it was a huge positive improvement. And yet, it's only made possible because Uber has a social credit score. If your Uber rating goes too low, you'll get kicked off the platform. And so we're fine with social credit scores. It's when you only have one and don't have an option and it can follow you across all these different verticals that it becomes a problem.

Gus Docker: Do you imagine that because of rising danger in the world, you talk about the externalities from the widespread implementation of AI all across society. Because of those dangers, those externalities, you will either use Uber or whatever service or you kind of can't participate in society. Do you imagine increased pressure in that direction?

Samuel Hammond: It does seem to be a longer term trend. I don't know if AI will accelerate it. I have another series of essays that I call separation anxiety. And it's a reference to the fact that in insurance markets, there's kind of two equilibria. There's the pooling equilibria where we're pooled together into one risk pool, and then there's a separating equilibria where the insurance pool unravels and we break up into, the great power insurance for senior citizens who never had an accident and stuff like that. And it turns out that insurance markets are competitively unstable, that without government regulation or social insurance, that insurance markets will naturally tend to unravel because of adverse selection into the high risk people being in one pool and the low risk people being in another pool. And it turns out you can sort of use that as a mental model to look at other kinds of implicit pooling equilibria. Right? So within company wage distributions, often there is 20% of the workers who are doing 80% of the work, but they're pooled together under one wage structure, and that was sort of the dominant structure of the period of wage compression in the United States in the 50s and 60s. And once we had better monitoring technologies and we're able to tell who were the 20% that were doing 80% of the work, it suddenly became possible to differentiate pay structure and a lot of the rise in inequality in the United States is actually between firm. So what happens is, Ezra Klein is the most productive whiz kid at the Washington Post, and he realizes, why don't I just go start my own website? Right? And so that dynamic sort of played out across a variety of domains leads to a world that, to the extent that these features are correlated, that does separate, right, where you have the one star Uber riders driving the one star Uber drivers, the drivers driving the riders, and people who have the five star Uber ratings and the perfect credit scores self sort into communities with other people with perfect driving records and perfect credit scores. And we see that to an extent already with the enclaves of rich zip codes with private schools and everyone is sort of self selected. AI could, it seems to me that AI would exacerbate that. I mean, at first blush just because it going back to the point about signal extraction, it can find all these different ways. You're a high risk type and I'm a low risk type and so forth that are probably latent in all kinds of data that we don't even need to give permission to the insurance company. They'll just, the same way that they use smoking or going to a gym as a proxy, there's all kinds of proxies they could use and likewise for employers and how they pay people.

Gus Docker: Society kind of runs on us not being entirely open and entirely honest all the time. Otherwise, you wouldn't be able to have kind of smooth social interactions and so on. Won't these norms be inherited by the way we use AI?

Samuel Hammond: Yeah. I think this is a really big issue. I'm a big fan of Robin Hanson, and a lot of his writing on social status and signaling is sort of presenting humans as basically hypocrites. We're constantly deceiving other people and we often deceive ourselves, so it's better to deceive others as the evolutionary biologist Robert Trivers has pointed out. So all the kinds of polite lies that we tell are, I think, critical lubricants to social interaction. And actually, it's good that there's a gap between our stated and revealed preference. I think a world where we all lived our stated preference could be hellish because we don't actually mean it. And AI has a direct implication on that because if I can have a pair of AR glasses on that will tell me if you're interested, if you're bored, if you're on a date and, are you really attracted to me? All that sort of polite veneer, that social veil could be lifted in a way that we'll probably want to coordinate to not do. Right? But, again, it's this Nash equilibrium where it's in my interest to know whether you're interested or bored. And so I'll want to have the glasses on, and my ideal world is where only I have the glasses and you don't. And the other way that our hypocrisy is being exposed and challenged is the need to explicate the utility function that we want these models to work under. We need to formalize human values if we want to align these models. And so then we have to be honest and open about the fact that our stated preferences probably aren't our true preferences. And that's a very challenging thing because it goes it cuts right to the nature of the human condition and involves topics that are intrinsically things that we lie to ourselves about.

Gus Docker: You have what you call a timeline of a techno feudalist future, which I found quite interesting. Yeah, it's great writing and it's very detailed. We don't have to go through all of its detail, but maybe you could tell the story of what happens in what you call the default scenario. This is the scenario in which Western liberal democracies are too slow to adapt to AI. And so we get something like a replacement of government services with more private services. What happens in the techno feudalist future?

Samuel Hammond: Right. And it sort of piggybacks on everything we've just been discussing. Right? And I don't want techno feudalist to carry too much of a pejorative. I'm sort of using it descriptively. And certainly, some people would prefer this world. So the example of Uber and Lyft displacing taxi cabs is sort of a version of this in micro, where we go from this regulated taxi commission to competing private platforms that use various forms of artificial intelligence and information technology to replace the thing that was being done by explicit regulation. And as AI progresses and both creates a variety of new negative externalities, whether it's suicide drones or the ability to spy on each other. There's going to be a demand for new forms of security and also kinds of opt in jurisdictions that tie our hands in the same way that we give up our phone before we go into the comedy club. And so I think this leads to a kind of development of clubs, the kind of club structure maybe at the city level as the vertically integrated walled garden that will police and build defensive technologies around the misuse of AI and at the same time provide a variety of new AI native public goods that are only possible once AI unlocks them. And it's easy to see how this could very quickly displace and eat away at formal government services, both because we saw it already with Uber, but also if you map that model to other areas of regulatory life. Does it make sense to have a USDA farm inspector? A human person has to go to a commercial firm and maybe only goes to that firm once every few years because there's so many firms and only so many people. And does a little checklist and says, oh, you're not abusing the animals and you got all the process in place and you get the USDA stamp of approval. Or does it make more sense to have multimodal cameras in the farm 24/7 that are continuously generating reports that throw up a red flag anytime someone sneezes on the conveyor belt. And to the extent that government is going to be slow at adopting that, will there be a push for the kind of Uber model of governance as a platform where you have the kind of AI underwriter, the consumer reports that sells these firms camera technology and the monitoring technology and builds their own set of compliance standards. And then you want to go to those firms that have the stamp of approval of the underwriter because it's much higher trust. It's sort of like the end of asymmetric information. And you can map that from food safety to product safety to OSHA and workplace safety. There's other parts of government that maybe just rendered completely obsolete, right? Once we have self driving cars that are 1000x more safe than humans, do we need a National Highway Traffic Safety Administration? Once we have sensors that are privately owned everywhere and can model weather patterns better than the National Oceanic Administration. Do we need a national weather service, or could we bootstrap that ourselves? And then once we have AI accelerated drug discovery, do we want to rely on the FDA to be a kind of choke point to do these sort of frequentist clinical trials that are inherently slow and don't capture the kind of idiosyncrasies and heterogeneity that could be unlocked by personalized medicine? Or do we move to an alternative drug approval process that is maybe nongovernmental but much more rapid and much more personalized?

Gus Docker: That's the overall picture. I'll just run through the timeline here, picking up on some of your comments that I thought were especially interesting. You write and this is in 2024 to 2027. You write that the Internet will become Balkanized and you will have it will become more secure and more private in a sense. Why does that happen?

Samuel Hammond: We're already starting to see this a little bit, right? Once people realize that the data that's being generated on Stack Overflow or Reddit or whatever is valuable for training these models, suddenly everyone's closing their API. And consequently, Google Search and the Google index have sort of started to degrade already. So I think that will continue for the kind of privatization of data reasons. Then we also think about how websites are going to handle sort of the growth of bots and catfishes and catfish attacks and cyber attacks and so forth, it makes sense that we're going to move from a sort of open, everything goes kind of Twitteresque platform to things that are much more closed because they require human verification and identity verification to sort of build the trust that you're talking to other people and not deepfakes. And then mid term, again, over this sort of 2024 to 2027 horizon, you can also start to see the emergence of intelligent malware, sort of modern AI native cyber attacks that could be devastating to legacy cybersecurity infrastructure in a way that, I talk about could harken back to the famous Morris worm that in the late 80s basically shut down the early Internet. Like, they literally had to partition the Internet and turn it off so they could rid the network of the worm. So for all those reasons, I think you start to see the Internet balkanize. And then particularly at the international level, we're already starting to see sort of the semiconductor supply chain become critical part of national security. The growth of the Chinese firewall, the European Union is going to have to have their own quasi firewall, and they kind of already do with GDPR and the EU AI Act. So the kind of nationalization of compute and telecommunications infrastructure that will take off once people understand both the security risks and the value prop of owning the infrastructure for the AI revolution.

Gus Docker: Yeah. In 2028 to 2031, you write about alignment turning out to be easier than we thought with the increasing scale of the model. That was somewhat surprising to me. Why does alignment turn out to be easier?

Samuel Hammond: And part of this is imagining a scenario where alignment is easy. So we can talk about what happens if alignment is easy. But I think there are reasons to think that the classic alignment problem will be easier than people think. I think that some of the early intuitions about the hardness of the alignment problem were rooted in a view of maybe AI turns out to be a very simple algorithm rather than a deep neural network that achieves its generality because of its depth. Clearly, the kind of value, I forget what Eliezer Yudkowsky called it, but the there's a value alignment problem where, how do we teach the model our values? But that part of the alignment problem seems trivial now because our large language models aren't autistic savants. They're actually incredibly sensitive to soft human concepts of value and context. They're not going to have the paper clip maximizer sort of monkey paw kind of threat models don't really make sense in that world.

Gus Docker: But there's a difference between the output of the model and the weights or the what the model has learned. And so just because a model can say, it can say the right words that we want it to say, but what has it actually learned? We are not entirely sure. And so it has learned to satisfy human values to some extent, but has learned to want to comply with human value out of distribution, in other domains and in a deep sense, I'm not sure about that.

Samuel Hammond: No, I agree. So I'm sort of just laying some of my groundwork to explain my priors on this. No, I agree, reinforcement learning from human feedback is not alignment in the same way that, you could argue that the coevolution of cats and dogs with humans led to a kind of reinforcement learning from human feedback in their short run evolution that made them look appear as if they experience guilt and shame and these human emotions when, in fact, they're just sort of a simulacra of those emotions because it means that we'll give them a treat.

Gus Docker: But I've done plenty of episodes on deceptions in these models and so on. We don't have to go through that, but I just wanted to point out that, yeah, maybe there's some complexities there.

Samuel Hammond: So my first prior is that these models aren't autistic savants the way they might have been. The second is going back to universality. While it's true that it's possible through reinforcement learning from human feedback, for example, that you're not selecting for honesty or selecting for a deep take of honesty. But in the bigger picture, the intuition that these models are converging or convergent with human representations should give you some confidence that they're not going to be as alien as we think they will be. It's also useful input for thinking about interpretability. There's some recent work showing that, discussing sort of representation interpretability where instead of trying to interpret individual neurons, you interpret sort of collections of neurons and circuitry through sort of human interpretable representations. And one of the lessons of universality is that some of these high level human concepts like happiness or anxiety, these seem like vague psychological abstractions that there's no way they can correspond to the micro foundations of the way our brain works. But in fact, they may actually be very efficient low dimensional ways of talking about what's happening in our brain. And then the third thing is I think that I just have seen, my sense is that the work on interpretability is actually making some good progress. Whether it can scale is another question, but I think we'll get there. In my timeline, I talk about sort of AGI level models within the human emulator plus domain. I do later on talk about superintelligence emerging maybe in the 2040s, and that's another story. Right? And so I think some of this stuff maybe goes out the window if we have models that are bigger than all the brains combined and have strong situational awareness, but I don't think that happens this decade. Certainly not with the current way we're building these models. With the way we're currently building these models, I think it comes much closer to a simulacrum of human brain.

Gus Docker: Got it. In 2036 to 2039, you talk about robotics being solved to the same extent or maybe even in the same way as we are now solving language. I found that super interesting. Explain to me why would robotics suddenly or quite relatively suddenly become much easier? Roboticists have been fighting for decades to get these models to walk relatively unencumbered and it's been an uphill battle. Yeah. Why can we solve robotics in the 2030s?

Samuel Hammond: This may end up happening sooner than I project, but I mean, if you look at LLMs, one of the stylized sort of trends with large language models is that natural language processing went from being this study of how to make machines understand language, went from being a dozen different subdisciplines. You had people working on parsing, people working on syntax, people working on semantics, people working on summarization and classification. And these are all different research directions. And then along comes transformer models and it just supplants everything. And LLMs can do it all. And I think robotics is sort of still in that ancient regime where a lot of what Boston Dynamics does is ad hoc control models, analytically solvable differential equations, different kinds of object recognition modules and control action loops and so forth. And so it's still in that early NLP phase where they have 12 different subdisciplines and they're sort of mashing them together. And, of course, you get something that's not very robust. I think we're already starting to see that paradigm shift to end to end neural network trained models like, Tesla, for instance. I think one of the reasons why Tesla cars had a sort of temporary decline in performance was because they were undergoing the transition from these ad hoc lane detectors and stop sign detectors and stuff like that to a fully end to end neural network transformer based model. And that turns out to be a much more robust way to train the model because stop signs look different in different countries and maybe a stop sign isn't the thing you care about really, so on and so forth. And so I think the transformer sort of scale deep learning revolution is only now coming to robotics, and people in that field are a little bit cynical because they're used to relatively small RL models thinking that the fit with actuators and some of the hardware is a really challenging problem and also believing that we don't have the datasets for it. But then you look at, those recent RoboDog that you may have seen on Twitter, a fully open source robot model for a Boston Dynamics style dog. It was trained on H100s, 10,000 human years of training and simulation, and then some fine tuning on real world data. And they have a very robust robot control model that you could plug into all kinds of different form factors and have something that can hop gaps and climb stairs and do all the things that Boston Dynamics robots don't do very well outside of their distribution.

Gus Docker: Do you think we will have a general purpose algorithm that we can plug into basically arbitrarily shaped robots that can then navigate our apartments or our construction sites or maybe our highways? That's an interesting vision, I think. Why is it that we achieve this level of generality?

Samuel Hammond: If you look at humans, humans are very good at, if we've suffered an amputation or you have to go through physical therapy, and it's not easy necessarily. But humans are able to adapt to different kinds of physical layouts of their body. And I think there will be a trend towards unified robotic control models that aren't super tailored to two legs and two arms and so on and so forth. Once you've installed it through a little bit of in context learning or fine tuning or reinforcement learning, adapt to that particular form factor. And this will parallel the kind of pre trained foundation model paradigm that is currently taking place in LLMs where you have the really big foundation model that can sort of do everything reasonably well, and then you can fine tune it beyond that.

Gus Docker: If we get to the 2040s in your timeline, you talk about massive amounts of compute being available. You talk about post scarcity in everything except for land and capital. And then you also talk about the development potentially of superintelligence at that point. What happens there? Who is in control of the superintelligence, if anyone?

Samuel Hammond: Yeah. This is sort of where I start to get a little bit tongue in cheek, but, I first of all, I talk about how, I tend to think that, once we have exascale computing and I think DOE just built their first exascale computer, and maybe it was private company, but we have one exascale computer in the world. But the 2040s, they'll be commonplace. And if we ever worried about sort of controlling the supply of GPUs, I don't know exactly how much compute will be on our smartphones, but it will definitely be possible to train a GPT-5 model from your home computer. And so any kind of AI safety regime that we built today doesn't take into account that falling cost of compute will probably break down. And therefore, amid this broader sort of fragmentation of the machinery of government, the state, I expect more and more government functions to be offloaded into basically private cities, HOAs, gated communities. And likewise with the Internet, I expect more and more of our sort of permissioning regime for new AI models and deployment to shift to the infrastructure layer where telecommunication providers will be monitoring network traffic for unapproved AI models and so forth, and we'll have Chinese style firewalls, but that are specific to a particular local area network. And at that point, the world looks, the United States, where this takes place, looks more like an archipelago of micro jurisdictions. I tend to think that a post scarcity political economy looks a lot like the Gulf States, Gulf State monarchies. Right? Because Gulf State monarchies are basically living post scarcity. Right? They have a spigot of oil they can turn on, and then they can go build mega projects in the desert. And they have infinite labor because they can just import guest workers. And so you end up with this, but if we can't have a Gulf State Monarchy in the United States, instead we have a bunch of micro monarchies dotting the country. So I sort of jokingly say, who's going to stop the free city of California that's home to all the trillionaire ML engineers and tech founders from the decade prior from plugging in their humanity sized supercomputer into a fusion reactor and turning it on.

Gus Docker: Yeah. And this is really the kind of endpoint of the discussion or your main point of institutions being eroded and then afterwards being unable to respond to strong AI.

Samuel Hammond: Yeah. And leading up to this, it sounds like a scary dystopian type of thing. It doesn't have to be. Right? Uber is not dystopian. Airbnb is not dystopian. Private airports in other countries are way better than the public airports in the United States. So privatization and the sort of techno feudalist paradigm doesn't have to be bad, But what it is is more adversarial. Right? And people have sometimes speculated, what, did the crumbling of the Roman Empire was a kind of prerequisite to a renaissance? Right, because it allowed for these principalities to sort of compete and to get the Florentine, creativity and so forth. I think the next couple decades could similarly be a renaissance for science and technology and for understanding the world. But it's probably a renaissance because we'll be moving into a much more competitive adversarial world where these city states and so forth will be hard to coordinate. And so to the extent that there are still these meta risks where we would value some large scale intra and international coordination, like peace treaties and so forth, the disintegration of the United States where this revolution is occurring would be bad for that.

Gus Docker: You talk about or you hint at an alternative path. What we've been talking about, your timeline here is the default path. You hint at a path where we have something you call constrained Leviathan. What is constrained Leviathan?

Samuel Hammond: It's limited government.Samuel Hammond: (1:59:16) Right? So this is Daron Acemoglu's word for it from The Narrow Corridor. And if you trace the rise of what we associate with liberal democracy, it is part of a particular technological equilibrium, in particular, an equilibrium that favored centralized governments with impersonal rule of law and impersonal tax administration and so on and so forth. So we associate today libertarians with being anti-government, but the basic idea of liberalism is actually associated with strong government, a strong impersonal government that can impose the rule of law. And so if we want to maintain that kind of equilibrium in a world where AI is diffusing on the society level faster than it is on the state and elite level, then we want to accelerate the diffusion of AI within government. And there's obviously low hanging fruit. We talked about how bureaucracies are basically flushing APIs. Even today, I have a friend at the FTC, the Federal Trade Commission, they have a 30 person team that is part of the healthcare division and they're in charge of policing the entire pharmaceutical industry in the United States for competition. His day job right now looks like manually reading through 40,000 emails that were subpoenaed from a pharma CEO, right? And today, you could take those emails and put them into Claude 2 or something like it with a big context window and ask, "find me the 5 most egregious examples of misconduct," and it would do that. It might not be perfect, but it's a hell of a lot more efficient than reading through them manually. And obviously, big law is going to be doing that, and the pharma CEO and his personal attorneys will be doing that conversely. To maintain our state capacity in the face of AI is to run in this arms race. And you can liken it to what in evolutionary biology they call the Red Queen dynamic, which comes from Alice in Wonderland where the Red Queen tells Alice that sometimes you need to run just to stay in place. And so I think our government needs to be adopting this technology as rapidly as possible so that they can basically tread water. And that means both diffusing it in existing institutions, but also being open to radical reconfigurations of the machinery of government and addressing some of those firmware level constraints that we talked about, whether it's the lack of a national identification system or the outdated information technology infrastructure or the accumulation of old procedural methods of governance.

Gus Docker: (2:02:00) One focused way of doing this is what you've called for in a Politico article, which is a Manhattan Project for AI safety. First question here, would it be better to call it Apollo Project as opposed to a Manhattan Project? I mean, the Manhattan Project created some pretty dangerous weapons whereas the Apollo Project might have been more benign.

Samuel Hammond: (2:02:22) I mean, what the Apollo Project and the Manhattan Project have in common is that they came from an era of US government where we still built things, where we still had competent state capacity, where we still had a lot of in-house expertise and we weren't saddled with all these constraints. So today, we couldn't go to the moon in 10 years. NASA couldn't. SpaceX can. And so our modern Apollo Projects are being done by the private sector through competitive contracts. And so one of the messages of my piece on the Manhattan Project is to say, the reason I make this analogy is not just because AI is an Oppenheimer-like technology, but also because responding to it will require a throwback to those kinds of institutional forms where we gave the people at the top a lot of discretion and sort of gave them an outcome and let them solve for that outcome without having much prescriptive rules about how to solve for that outcome. And then the second reason to make the analogy is OpenAI and Anthropic, they both have contingency plans for developing AGI and having runaway market power. In the case of OpenAI, it's their nonprofit structure. In the case of Anthropic, it's their public benefit trust. They both are envisioning a world where they could potentially be the first to build AGI and become basically trillionaires. And so at that point, they need to become basically governed by a nonprofit board. And that's not where progress ends, obviously. There's going to be continued research. It would make sense for the US government to step in and say, let's do this as a joint venture. We're no longer competing. In fact, the basic structures of capitalism and market competition are starting to break down. Let's just pull this together into a joint venture, study the things that require huge amounts of capital that the private sector doesn't have but the government can. The US government spent $26 billion on the Manhattan Project in today's dollars. When you think about the financial resources of nation-state actors to put behind scaling, it's nothing like what Microsoft or Google have. When's our first $200 billion training run? What kind of things can come out of that? I think that's something that you want to do with the defense department's involvement and working with these companies in a joint way through secured data centers and doing gain of function style research that really is dangerous and more Manhattan Project than Apollo Project.

Gus Docker: (2:04:59) What would be the advantages here? We would be able to spend more to slow down capabilities research and spend more of the resources on, say, mechanistic interpretability or evaluations or alignment in general, because now the top AI corporations have kind of combined their efforts under one government roof?

Samuel Hammond: (2:05:22) Yeah. Yeah. And in my vision, they were still allowed to pursue their commercial verticals, and I have an extended version of the proposal where I talk about needing biosafety style categories for high risk, medium risk, and low risk styles of AI that closely parallels what Anthropic recently put out with their recommendations for a BSL categorization of AI research. So I'm really talking about that BSL-4 lab and beyond style stuff. And some of that stuff will be to accelerate alignment and durability research to sort of do versions of the OpenAI Superalignment Project where they're dedicating 20% of their compute to study alignment. Another part of it will be to forestall competitive race to the bottom dynamics so that they can coordinate and not violate antitrust laws. And then the third thing is the gain of function stuff that we really only want to be doing with very, very strict oversight compartmentalization, pooling of talent and resources so we can share knowledge on alignment and safety. But then also because government has this huge spending power relative to the private sector, anytime you build a supercomputer, you're basically borrowing from the future. You're trying to see what the smartphones 20 years from now will be capable of. And so if we want to sort of get ahead of the curve and see where scaling is leading, then I think governments are really the only actor that can waste a bunch of money basically scaling up a system and seeing what comes out of it.

Gus Docker: (2:07:06) Yeah. When we talk about gain of function research in AI, it's an analogy to the gain of function research that's done on viruses in biolabs, done for AI models. This could be experimenting with creating more agentic models or inducing deception in a model and planting it in a simulated environment, seeing what it does or enticing it to acquire more resources. But again, if this is even possible in a safely constrained simulated environment. And this is the type of research that we could do in this Manhattan Project, this government lab, because we would have excellent cybersecurity and secure data centers and the combined efforts of the most capable people in AI research.

Samuel Hammond: (2:07:58) If you've watched Oppenheimer, the movie, a lot of that revolved around suspicions of communist spies and so on. And we really don't have great insight into the operational security of the major AGI labs. And that's something that bringing in-house at the defense department would—they would necessarily have to disclose everything they're doing, but also hopefully beef up their operational security.

Gus Docker: (2:08:26) Yeah. They're kind of stuck with a startup mindset, but they're not developing a startup product. They're developing something that, in my opinion, could be more dangerous than the average startup.

Samuel Hammond: (2:08:39) Yeah. And Dario Amodei has said as much that we should just assume that there are Chinese spies at all the major AI companies and at Microsoft and Google.

Gus Docker: (2:08:47) When we think about gain of function research in AI, how do you think about the value of gaining information about what the models can do and what the models can't do versus the risk we're running? It would be a tragic and ironic death for humanity if we experimented with dangerous AI models to see whether they would destroy us and then we hadn't constrained them properly and they actually destroyed us. So how do you think of that tradeoff between gaining information and avoiding lab leaks?

Samuel Hammond: (2:09:23) Yeah. Hopefully, lab leaks are less likely than in the biology context where getting a little bit of blood or urine on your shoes as you walk out the door—it's a difficult thing to talk about in part because we just went through a pandemic that very probably was caused by a BSL-4 lab leak. And so one saving grace is that AI models don't get caught in your respiratory system. And so, hopefully, there's forms of compartmentalization that are much, much more robust than in the biology context. And to the extent that this research is going to be done anyway, it would be much better to move it offsite and hopefully in a way that facilities are air gapped and so forth rather than what Microsoft is doing right now. Microsoft just recently announced their Autogen AI, which are sort of agent-based models, very similar to AutoGPT, but at work. And they're doing this through Creative Commons, totally open source framework. All this capabilities work is gain of function research. Where we draw the line between doing things that are intentionally dangerous or doing things that are dangerous, but we're kind of pretending they're not, is hard. I do think there's—and Paul Christiano has also agreed with this—threat models that would be valuable to be running in virtual machines to see if the AI develops awareness, situational awareness, and tries to escape, but it escapes into a simulated world that we built for it.

Gus Docker: (2:11:01) Okay. Let's end by talking about a recent critique of expecting AGI to arrive pretty soon, in a short time. This revolves around interest rates. And I guess the basic argument is or the basic question is if AGI is imminent, why are real interest rates low? I can explain it, but you're the economist, so maybe you can explain the reasoning here.

Samuel Hammond: (2:11:28) So it's really a question of how efficient are markets and how much foresight do markets have. We're coming out of a world of very low interest rates, of ultra low interest rates, near zero interest rates, and one way to think about that is there's a surplus of savings relative to investment. And so one of the reasons interest rates have been in secular decline is because populations are aging and so old people have a huge amount of savings built up, and meanwhile, we're going through this sort of technological stagnation. So the amount of savings relative to the amount of profitable investments was out of whack, and so that pushes interest rates down. In a world where AI takes off, it's a world where we have enormous investment opportunities, where we'll be building data centers left and right and we can't do it fast enough, where there's new products, new commercial opportunities left and right. And so you would expect in that world where the singularity is near, so to speak, to be one where the markets begin forecasting rapidly rising interest rates because the savings to investment balance is starting to shift. And in addition, there's a long run stylized fact that interest rates, real interest rates track growth rates. And so if GDP growth takes off, you'd also expect at least nominal rates to also take off. And so some have argued that looking at current interest rate data like the 5 year, 10 year, 30 year treasury bonds, that the markets are not predicting AGI. Two responses to that are, one, first of all, interest rates are up quite a bit. Nothing's monocausal. There's lots of confounding factors. Is this, to some extent, the market anticipating an investment boom? Maybe they're not anticipating full AGI, but they're seeing the way LLMs are going to impact enterprise and sort of taking some of that in. And then the second piece would be, okay, to the extent that they're not pricing in AGI, how much foresight do markets have anyway?

Gus Docker: (2:13:27) Before we discuss market efficiency, I just want to just give a couple of intuitions here. If AGI was imminent and it was unaligned, say, and it would destroy the world in 5 years, well then it doesn't make a lot of sense to save money. Similarly, if AGI is about to explode growth rates, well then a lot of money will be available in the future. You're about to become very rich, so it doesn't make sense to save a lot now. And the pool of available savings determine what's available for lending, which determines interest rates. But let's discuss whether markets then are efficient on this issue or to what extent they're efficient.

Samuel Hammond: (2:14:15) Right. So this is the efficient market hypothesis, which comes in strong and weak forms. So the strong form of the efficient market hypothesis would say that markets aggregate all available information and are our best point estimate of anything we care about. The weaker form, which I think is more defensible, is that markets can be wrong, but they can be wrong longer than you can be solvent. Right? And so you can try to short a company like Herbalife, famously, there's a big short position on that because Herbalife sort of looks like it's a multilevel marketing Ponzi scheme, but yet the hedge fund that did that lost several billions of dollars before they ended their position because the markets stayed irrational longer than they could stay solvent. The second factor is the weaker versions of the efficient market hypothesis are sort of based on a no arbitrage condition. Right? They say markets are efficient only insofar as you can arbitrage an inefficiency, right? And so, you look at some prediction markets, for example, they'll often have very clear inconsistencies across markets that look like they're irrational. But then you realize, oh, I can only make $7,000 total on the website and there are transaction fees and I have to—there's work involved. And so if the market isn't very deep or liquid, there may be inefficiencies that exist not because the market's inefficient, but it's as efficient as it can be under the circumstances. And when it comes to AI, how do you arbitrage? I've been thinking for a while now that Shutterstock, their market cap should be collapsing. Right? Because we have image generation that is proliferating. And, yes, people will make the argument, though, Shutterstock has all this image data. They could build a better image model. It seems like it's cannibalizing their business. It's sort of turning a moat into a commodity. And yet Shutterstock's market cap has basically held constant throughout this recent birth of image generation models.

Gus Docker: (2:16:25) What if you borrow a lot of money cheaply and then put it into an index of semiconductor stocks or just IT companies in general, even just general S&P 500 today? Would that be a way of arbitraging this AGI forecast?

Samuel Hammond: (2:16:41) Yeah. I would say if you have short timelines, you should be putting a lot of money into equities.

Gus Docker: (2:16:47) This is not financial advice, would you say?

Samuel Hammond: (2:16:49) Right. And that's—and I mentioned earlier that Paul Christiano has said in interviews that he's twice levered into the stock market. He basically owns a bunch of AI exposed companies, and he's borrowed enough money to double his investments. So that's putting your money where your mouth is. When you look at market behavior over the long stretch of time, markets didn't anticipate the internet very well. There was a short run bubble that led to a boom and bust of dot-com stocks. But in terms of the real economy, the internet just kept chugging along and kept being built out. And eventually, a lot of those investments ended up paying off even if you rode through the bubble. Markets are made up of people. Some of the biggest capital holders in the markets are institutional investors, pension funds, life insurance companies, governments, like Saudi Arabia or the Norwegian pension fund. And often, these are making safe bets. They're not taking very heterodox views on markets. And so as a result, markets can be a little bit autoregressive. They're a little bit biased to the past, past is prologue and prone to kind of multiple equilibria where there's two prices that Shutterstock can be. Shutterstock could be a $50 stock or it could be a $0 stock, and at some point, the market will update and will undergo the great repricing, and all these asset prices will flip in relatively short order. The efficient market hypothesis has to be false or else we wouldn't have Silicon Valley. Right? We wouldn't have founders. We wouldn't have Elon Musk. Right? So I would just say the markets are wrong. And partly, they're wrong because to be right would require having a bunch of relatively bespoke and kind of esoteric priors about the direction of technology that are only now just sort of percolating into the mainstream.

Gus Docker: (2:18:53) Yeah. And the big capital allocators can't really respond to because they're risk averse.

Samuel Hammond: (2:18:59) Exactly. That doesn't mean Renaissance Technologies won't respond to it, but they're not going to move the market.

Gus Docker: (2:19:05) Samuel, thanks for this conversation, and I've learned a lot.

Samuel Hammond: (2:19:08) Thank you.

Gus Docker: (2:19:09) It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.