Full Transcript

Transcript

Tim Duignan: (0:00)

Finding efficient ways of keeping track of the important information and losing the unimportant information is just a central problem in a lot of physical modeling, and I think AI and machine learning algorithms are just very good at doing that.

Jonathan Godwin: (0:13)

The thing that completely blew my mind was training on small inorganic crystals, like 20-atom systems, to then simulate a protein through out-of-the-box generalization. That's telling you that we're learning something really fundamental at that small scale, which I don't think anyone had ever expected.

Nathan Labenz: (0:32)

The hope is that if you can simulate a ton of things, start to make important and new discoveries and find new things out just by looking at them, which is very exciting.

Jonathan Godwin: (0:39)

By the time we get to making a decision about what to make, we've answered 90% of the questions that we need to in order to feel confident that we're going to have that sort of material.

Tim Duignan: (0:50)

The real key challenge there had been these potassium ion channels, which no one has really been able to fully understand, unfortunately, using experimental techniques or traditional computation. And in fact, we don't even know some of the most basic questions about it.

Nathan Labenz: (1:06)

Hello, and welcome back to the Cognitive Revolution. Today, my guests are Jonathan Godwin, founder and CEO of Orbital Materials, which is pioneering the application of AI to materials science, and Tim Duignan, who was previously here to discuss his work on the simulation of electrolyte solutions, and who's since joined Orbital Materials as a researcher. Material science underpins virtually every aspect of modern life. From the semiconductors that power our devices to the batteries and solar panels driving the clean energy transition, advances in materials have been at the heart of human progress for the last century at least. The challenge has been that discovering and developing new materials has always been painstakingly slow, traditionally relying on trial and error and scientists' hard-won intuitions developed over decades. And more recently, with the shift to computer simulation, still requiring huge computing power to simulate even small molecular systems for short time intervals. Orbital Materials aims to dramatically accelerate this process with, of course, AI. Their immediate focus is on developing novel materials for data centers, both to improve efficiency and to capture carbon emissions, but their technical breakthroughs could unlock advances across clean energy, electronics, medicine, and beyond. Their technical approach is really quite fascinating. Using an architecture called message-passing neural networks, which are trained on small crystal structures and which, because they don't use positional embeddings like large language models do, are capable of scaling up indefinitely with computing power. They can design new materials with specific target properties via a diffusion process and also predict the forces between atoms orders of magnitude faster than numerical methods can, which allows them to simulate larger systems for longer. They recently demonstrated the power of this approach by simulating a potassium ion channel, a critical protein that controls electrical signaling in our cells by selectively allowing potassium ions to pass through cell membranes. Despite its importance in everything from heartbeats to brain functions, fundamental questions about how this channel works have remained unanswered for decades. And while Tim's recent work still needs to be experimentally confirmed by the broader research community, his simulations were able to show a level of detail in the mechanism that was never before seen, and which does help explain previously inexplicable data. The implications for biology and medicine are significant, but perhaps more important still, this work illustrates a critical phenomenon that we are seeing time and again as AI is applied to the different branches of science. Namely, that neural networks seem to have the ability to develop a sort of intuitive physics in virtually any problem space. Just as humans can catch a ball without explicitly calculating its trajectory, these AI systems are developing efficient shortcuts for predicting complex physical phenomena. Whether that's material properties, protein folding and interactions, weather forecasts, single-cell transcriptomes, or even the evolution of human brain states. For me, this is the clearest reason to believe that superhuman intelligence is not just possible, but increasingly likely. For human scientists meanwhile, this means a shift away from hypothesis generation and toward more validation and implementation work. And while that might mean lower job satisfaction, the potential to dramatically accelerate scientific progress and more effectively address critical global challenges, for me, makes it a worthy trade-off. And again reminds us that we might soon need to look beyond our work for meaning. As always, if you're finding meaning in the show, we'd appreciate it if you'd share it with friends. You can also write us a review on Apple or Spotify, and we love to read your comments on YouTube. We value your feedback and suggestions too, so we encourage you to leave them either via our website, cognitiverevolution.ai, or you can always DM me on your favorite social network. For now, I hope you enjoyed this look at how AI is transforming material science, and by extension, how it might fundamentally reshape and accelerate scientific progress in general, with Jonathan Godwin and Tim Duignan of Orbital Materials. Jonathan Godwin, founder and CEO of Orbital Materials, and returning guest and now researcher at Orbital Materials, Tim Duignan, welcome to the Cognitive Revolution.

Jonathan Godwin: (5:21)

Thanks for having me.

Nathan Labenz: (5:23)

I'm excited for this, guys. I'm on quite a journey to try to figure out what is going on at the intersection of AI and science. And obviously, there's a lot of different sciences and a lot of different intersections and a lot going on. Life right now for me is one big crash course. You guys are advancing the frontier when it comes to material science and the use of AI in material science, and I'm excited to get into all that. I thought maybe just for starters, because listeners to this feed are obviously plugged into the AI moment, but probably most don't have much background in material science. Maybe just give us the basic foundational pitch for materials. Why do they matter so much? Maybe a couple highlights from recent times where an advance in the materials directly led to a change in the way we live, just to get the juices flowing and get people motivated to understand all this stuff.

Jonathan Godwin: (6:18)

The most obvious one for people who are into AI is semiconductors. You have this wafer, the silicon wafer, and you etch a bunch of stuff, your circuits onto the silicon wafer for your chip. But of course, those etches are incredibly thin, incredibly small. And if a piece of dust or some debris gets stuck in that etch, then your whole chip fails. So you need to find materials that don't conduct electricity because conducting electricity across the etches in your wafer will screw up your chip. So it can't conduct electricity, but it's got to be small enough to fit in between all of the different bits in your wafer in order to protect your chip from any sort of debris or pollution that comes on. So these things are called low-K materials. And without significant advances and improvements in the performance of these materials year-on-year, you didn't get more transistors on your chip and you didn't get AI. So material science underpins literally everything in our lives. The entirety of the Silicon Valley revolution has given rise to the tech industry in the West is based upon material science. And material science is the original tech startup. And we've just kind of forgotten that these companies built a lot of the industry in America, in the UK where I'm from, but all around the world. And I think the 21st century is going to be any different from the 20th century. The 20th century was the century of material science. The 21st century is going to be even more so. We're going to have more chips. AI is even more important. We're going to go to Mars. We're going to have to have incredible spaceships built out of new materials to do that. We're going to have HoloLens and virtual reality. We're going to have new optical materials to do all of that. So I think material science is the fundamental missing piece in achieving the science fiction future. And AI is our most powerful tool to be doing that. So that was just one example of all the things. But Tim, you've been working on materials for longer than me, so you must see many more.

Tim Duignan: (8:30)

No, I think that's an excellent one. If you think about the ideological factor about Bell Labs and the birth of the transistor, it's just all material science, trying different elements. But the other big one obviously is climate change, where essentially all of the tools we need to fight climate change are material science problems, broadly defined. And the most successful tools we've built so far have come out of material science. So batteries and solar panels leading to massive reductions in CO2 came from breakthroughs in material science. It's so universal that I often like to play the inverse problem. Name a problem, and then I can show you how it's related to material science in one way or another. It's just at the foundation of so many things.

Nathan Labenz: (9:14)

So maybe a little bit of the status quo would also be helpful in terms of how material science advances today. We've done versions of this for biology, for example, and have contrasted somebody sitting there with a pipette and what that throughput can look like. And obviously, some of that is starting to get automated in the physical world, but then we're starting to move more and more of that into simulation and even into AI models. You guys are basically doing the same thing as I understand it. I saw the motto on the website, "engineer in silico, materialize in reality." But for folks who don't know, before we get to that engineering in silico, what is the sort of life of a material scientist like today? Obviously, it's such a big field. There's no single answer to that. But just a little flavor for what it takes and what the experimental cycle looks like to make advances pre-AI.

Tim Duignan: (10:12)

Sure. So for an experimental material scientist, there's still a huge amount of trial and error and guesswork involved. If you talk to an average experimentalist trying to make a new material, they'll have hypotheses, they'll have ideas about what's going on and what they want to do, but there's a huge amount of luck involved, trial and error, and intuition as well, built up over years of what's going to work and what's not. I like to make the analogy to bridge building 100 years ago, where we didn't have highly accurate models of what bridges were going to be stable and what weren't. And so there was a lot of guesswork and intuition about how you would build these bridges, and things built on experience. And they could build incredible things, obviously, but it wasn't nearly as efficient. They couldn't push things to the limit, and they couldn't do it nearly as quickly as we can today, where to design a modern bridge, you would plug it into some kind of computational modeling software. And every bolt and beam, you would know the forces and stresses and tensions. You would know the limits of that material, and you could design everything really optimally. So if you look back at the Brooklyn Bridge, for instance, massively overdesigned versus what it needs to be because they needed to make it safe so it wouldn't fall over. And you did used to get new bridges falling down. There's a famous one where it hit a resonance because they hadn't tested it computationally, and it collapsed after they built it. Whereas today, we model these bridges incredibly carefully. We know before we build them exactly what they're going to do. I think we want to transition material science to that kind of world where we know in advance if we can make this material, we're very confident it will have the properties we want, and it will do the things we want. We also want to be able to answer questions much more quickly that material scientists have. So they might say, oh, this isn't working for some reason, I'm not getting the yield I expected, or the performance is bad. We want to be able to go to the computer and say, okay, understand this from the ground up and say, okay, here's exactly what's going wrong, and here's some suggestions on how to fix it, and here's what's actually happening at the molecular scale, which is a big problem often. We can't see what's going on.

Nathan Labenz: (12:05)

So where are we on the path to that dream? And that's maybe a great opportunity to introduce the company and tell a little bit of the founding story if you want and give the big vision.

Jonathan Godwin: (12:16)

Yeah. So Orbital came about when I was working at DeepMind and leading a team building large-scale AI models for material science. And I thought it was going to take us a decade to reach the level of performance we were able to reach in two years. The speed at which this field is changing is exactly like every other area of AI. It's just changing and developing at an unbelievable pace, and new capabilities that we had dreamed about are now reality. Some of the things that Tim, I'm sure, is going to talk about in more detail later on, they just weren't possible on the world's best supercomputers just 10 years ago, and we're now doing them on our laptops. And so the goal of our company was to take and continue to develop those models and those techniques, but combine that with a wet lab, with a team of extraordinary experimental material scientists to translate those AI capabilities into real materials. And we felt that by creating a team that had both of those skills, we were going to be able to move faster than anybody else was, and bring those materials to market and commercialize them better. And we were going to ground the AI that we develop in the real problems and bottlenecks associated with making breakthroughs in material science. And that team wasn't going to exist at DeepMind. It wasn't going to exist at Google. Those are AI companies. It wasn't going to come out of a traditional chemistry or material science company. They've been saying no to computational tools for the last 25 years. They're not going to suddenly turn around and be building frontier tech. So it had to be a new company if we wanted to achieve that goal. So I think we're so much further along than we were just a couple of years ago. The things that we're doing now are completely different. I think it's only going to be a matter of a couple more years where we're achieving the dream of new computer-designed materials, just get a GUI in front of you with an LLM copilot, and you'll get to a new battery in a couple of iterations just on your laptop, and have a real high confidence that when you go and make that, it's going to have those properties. And all of the things that I spoke about right at the beginning, spaceships, better chips, better semiconductors, more GPUs, more AGI, solving climate change. All of that is going to happen through the technology that we're building.

Nathan Labenz: (14:53)

One thing you said that caught my ear that I want to dig into for a minute is this idea of simulation. So it seems like we have kind of three modalities for pushing things forward. The original is hands-on with actual physical materials, doing stuff, measuring properties, and ultimately everything still has to ground out to the ability to do that successfully to achieve the results we want to achieve. And then there's been this sort of middle time of simulation. And you said that some of these things, we couldn't possibly simulate this even on the world's greatest supercomputer. And now we've got this third wave, which is the AI wave. And this was a big theme of conversation I had with Tim last time when we were talking about the crystallization discovery, which was basically that what distinguishes the AI era for this sort of work from the simulation era, maybe above all, is it runs orders of magnitude faster. So I've been really chewing on that quite a bit, and I'm coming to a worldview that basically amounts to a belief that today's architectures seem to be capable of learning what I might describe as an intuitive physics in basically any space that you might want to have them learn that. We obviously have an intuitive physics where we can play sports and react to things in sub-second time, which if we were actually using our feeble brains to do all the calculations and simulate step by step, we would never be able to catch the ball. But we're quickly sort of grokking the situation and not calculating in the way a Wolfram would suggest that we might have to. We seem to be taking a sort of intuitive shortcut, and it seems like that is popping up in all of these critical domains, whether it's protein folding or material science or solving the wave equation for a bunch of atoms in a configuration or whatever. Is that how you see it too? Would you edit my picture of that? Because that seems like a pretty foundational update to my worldview that I kind of want to go shout from the rooftops, but I want to make sure I have it right before I repeat it too loudly.

Tim Duignan: (17:08)

I think that's right. And I don't think we fully understand why yet, or at least I don't. But it's very exciting that this seems like a universal tool. It's revolutionizing climate simulations as well, it seems to be, and weather modeling as well. It's across-the-board physical simulation. My theory is it's something to do with dimensionality reduction, that often you don't need to treat every single part of a system. And so in our case, this is called coarse-graining in physics and statistical mechanics. There's great theories of it, Mori-Zwanzig. It's all based on Langevin dynamics, actually. So it's intimately related to what a lot of the AI algorithms are doing. But the basic idea is that you get these universal things across different scales, and they have repeating physical properties. And the key one is that you don't need to treat every single piece of the system to predict what it's going to do. So in our case, we're ignoring the electrons. In a lot of biology, you might want to ignore the solvent. And then when you get to larger scales, obviously, when you're looking at a ball moving through the air, it's just one object you're tracking, and you're ignoring this vast amount of other information going on. And so finding efficient ways of keeping track of the important information and losing the unimportant information is just a central problem in a lot of physical modeling, and I think AI and machine learning algorithms are just very good at doing that. But what do you think, Johnny?

Jonathan Godwin: (18:23)

Yeah. I think you're absolutely right. What's been extraordinary to me has been not just the kind of universality of the AI models that we use. So the stuff that we develop is very similar to the stuff that's used in weather forecasting, similar architectures, almost identical. But also just how much implicit physics knowledge can be contained in a very small amount of space. When you look at some of these world models, like Genie 2, you've seen that from DeepMind, or a couple of other startups have produced something similar. They have these single models that simulate physics, because they're simulating water dynamics and reflections of light in puddles. They understand gravity. They're able to deal with very small, tiny effects that just a couple of years ago would have taken really expensive computer rendering, CGI-type computer software in order to produce. But these only have, I mean, I don't know the precise number for these models, but these must only have a few billion parameters. These are going to be diffusion models. Diffusion models are not parameter heavy. They're very data intensive, but they require far less compute during your inference pass for learning these dynamics than a large language model does. So there's this implicit, as you say, this intuitive knowledge, because we know that the bottom-up calculation of all of these effects and the effects that we model when we think about atomic-scale quantum effects, which is the stuff that we really care about, calculating from bottom-up requires a huge amount of floating-point operations. So we're taking a lot of jumps. Whatever's going on, you can call it intuitive, but it's shortcuts that preserve the fidelity of that simulation in the ways that we care about. It is just quite extraordinary that we're able to condense that information in a way that has that level of fidelity into something just so incredibly small. I mean, the big competitor neural networks were a couple of years ago are tiny compared to our brain. They're tiny compared to the amount of compute we were throwing at it with bottom-up physics simulations. So it is quite incredible. And I think that intuitive physics explanation is a really nice way of thinking about what's going on because it's just in the same way that you get extraordinary sports people who can score incredible goals and do incredible feats, but not know any maths. That's exactly what these AIs are doing when they're doing these simulations.

Nathan Labenz: (21:07)

Hey, we'll continue our interview in a moment after a word from our sponsors.

Nathan Labenz: (21:11)

In business, they say you can have better, cheaper, or faster, but you only get to pick two. But what if you could have all three at the same time? That's exactly what Cohere, Thomson Reuters, and Specialized Bikes have experienced since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing-fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high-availability, consistently high-performance environment, and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50% less for compute, 70% less for storage, and 80% less for networking. And better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is the cloud built for AI and all of your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com/cognitive. That's oracle.com/cognitive.

Nathan Labenz: (22:21)

It is an interesting time for business. Tariff and trade policies are dynamic, supply chains are squeezed, and cash flow is tighter than ever. If your business can't adapt in real time, you are in a world of hurt. You need total visibility from global shipments to tariff impacts to real-time cash flow, and that's NetSuite by Oracle, your AI-powered business management suite trusted by over 42,000 businesses. NetSuite is the number one cloud ERP for many reasons. It brings accounting, financial management, inventory, and HR all together into one suite. That gives you one source of truth, giving you visibility and the control you need to make quick decisions. And with real-time forecasting, you're peering into the future with actionable data. Plus with AI embedded throughout, you can automate a lot of those everyday tasks, letting your teams stay strategic. NetSuite helps you know what's stuck, what it's costing you, and how to pivot fast. Because in the AI era, there is nothing more important than speed of execution. It's one system, giving you full control and the ability to tame the chaos. That is NetSuite by Oracle. If your revenues are at least in the seven figures, download the free ebook, Navigating Global Trade, 3 Insights for Leaders at netsuite.com/cognitive. That's netsuite.com/cognitive.

Nathan Labenz: (23:46)

Before we get more into the practical details of how you've gone about all this, one other philosophical question relates to, I don't want to be too literal about grokking, but there are these debates in language models where we get very confused. And you hear people say things like, well, sure, they might learn the concepts that we have learned and encoded in language, but they're not going to learn anything beyond that because how would they? They can only learn what's already encoded in language. And I sort of find discussions around that sort of topic can easily get circular. But then I move over to a different domain and we've now seen multiple examples of this out of the intersection of AI and biology where you can take interpretability techniques and apply them to a protein language model and find out that, I recently had a professor from Stanford, James Zhao on, and he was like, they discovered a new motif by looking at applying interpretability techniques to a protein language model. And so it seems like there's something there where it's pretty hard to deny that this thing has learned a concept that we did not know and it didn't get it from us, because in that case, it's just training on sequence data. But I don't know that I work that way. So this is where the analogy to intuitive physics maybe breaks down. I don't think, maybe I have, I just can't articulate it. I really have no idea, I guess, what's going on inside my brain. But it seems like there is some sort of conceptual grokking that is happening in, for example, the protein language models that are allowing them to do these shortcuts. It's these higher-order concepts. Is that also what you think is going on in material science? And do we know what those concepts are? I have a sense for a motif and an active site or whatever, but I don't even really know what those higher-order concepts would be in material science that these models might be grokking out of the raw data that they're trained on.

Tim Duignan: (25:55)

Yeah. I mean, one thing is I think that may be going on in scientists' brains, where a lot of science, I mean, people don't think of it this way, but it's really intuitive. It's really gut instinct driven. You rely on your gut all the time. And it's possible that even though we don't rationally use these ways of discovering things, that it could be that scientists are doing this intuitively. And definitely, in material science, I think that's very common. Call it intuition or chemical intuition is a very important concept and people rely on it. They can't often, they'll give you a story, but it's a very powerful tool. So it's very possible that the AIs are doing something similar.

Nathan Labenz: (26:35)

So what does that look like just to calibrate myself? Does a good material scientist just sort of put a finger to the wind and say, yeah, let's throw in a little more acid?

Jonathan Godwin: (26:45)

I've got a little story about this. I'm stealing someone else's story, but my answer is exactly what happens to semiconductors. People basically spend 30 years of their life on this very tiny thing. I was talking about etching before. It's like, what gases do you use to clean? You etch a silicon wafer, you need to degas it, you need to clean it with another gas. Which gases do you use? How do you change that slightly? How is it going to affect the silicon? People just spend 30 years of their life on each one of these. I think there are about 10,000 process steps or something in making a chip—the packaging, the etching, the lithography. These are craftspeople, artisanal. They've done so many experiments and gained such intuition that it's not written down anywhere. You can't go and learn this stuff. It's not easily deducible from first principles chemical knowledge, and it's passed down in this sort of master and apprentice type way. It's one of the reasons why people just can't recreate TSMC. However much money the US goes and throws at new chip fabs or China, however much espionage they go and do to try and build their own fabs in China, you can't beat TSMC because they've just spent all of that time and that intrinsic knowledge sits within their chemists and material scientists. So it really is this kind of deep sense of intuition. This won't work. This will work. One of the goals of AI is to break that, really, to say, well, AI should be able to, without having to go through all of that—ultimately, it's still physics. So AI should be able to get it right if they can get physics correct, and it should be able to democratize that level of knowledge. But yeah, it's like a sage person who's done this 30 years pronounces their views, and it's almost prophetically correct in many cases. You think, no, that can't be right, but it actually is. So my level of admiration for human practice and intuition, which is something that's not really captured in something like a reasoning model—this intuitive movement through the search space as scientists and chemists and material scientists—that has grown exponentially through my experience. We've got some incredible people that just know what will and what won't work.

Nathan Labenz: (29:21)

Tim, has your experience with chemists and material scientists been the same?

Tim Duignan: (29:27)

Yeah, I think so. And it's sad because it's a beautiful thing, right? This built up body of amazing knowledge, and it's sad to kind of be trying to replace it. But I think there are fundamental limits to the human brain, and we want to push beyond that basically. So we need ways of doing similar stuff computationally. And I think it's very exciting if we can start to get there. And then potentially as well, combining it with the human intuition is very powerful, as Johnny was saying, because they've obviously reached it from very different ways. They're going to be able to correct problems that each other makes. So I think it'll be a powerful combination when we can get them working together really well.

Nathan Labenz: (30:03)

Another thing that caught my ear was you said that the diffusion models are not parameter heavy, but they're data intensive. We talked last time with Tim about the training on molecular simulation data. So if I remember that accurately, it was like, it costs a lot of compute and I think it was 10 to the minus 15th second time intervals that were being simulated. And then you train the model on that and then you get a model that can do essentially the same thing, but orders of magnitude faster. AlphaFold, if I recall correctly, was only trained on maybe five figures, maybe in the low six figures worth of protein structures that were known at that time, which I think was basically everything that was known at that time. What does the data universe look like in a material science context? Because I can imagine it could be like a simulation thing or it could be a big catalog of all materials that we've characterized to date or maybe even both of those?

Tim Duignan: (31:06)

Yeah, this is a big issue in chemistry, right? For a long time, I was quite skeptical of AI for chemistry just for the problems I wanted. There's just vanishingly little data experimentally. I was interested in electrolyte solutions, and there's some databases out there, but there'll be a few parameters per electrolyte. And so I really thought, oh, this is going to be really tough to do anything. Now there's other areas of chemistry where there are large datasets, but in my particular area where I wanted to apply AI, I just couldn't see how to do it. Obviously, yeah, there's the PDB in biology with these huge datasets. But the thing that really clicked for me was, well, we can automatically generate these training datasets using first principles computation. It's kind of trivial. I realized I have huge datasets if you take into account the massive simulations I've been running based on first principles physics. These are generating millions of data points pretty trivially. Because the data is generated computationally, you can do all sorts of nice tricks like active learning and automatically generating the data in loops where you need it and things like that. So that's why I think first principles generated data is a big area of future growth in science.

Jonathan Godwin: (32:13)

Somehow you can think of this as like, where does AI generalize well? Because the data that you can generate quickly is often pretty small scale. Maybe I can give you an example. You are simulating what Tim's just simulated—a protein complex interaction with a metal. Now you're simulating that thing in isolation. It's a small part of something far bigger. And all of the water molecules that surround all of the rest of the cells or the rest of the proteins affect the dynamics of that in some small way. Now, hopefully, for the protein situation, it doesn't affect it so much that the protein stops functioning. But for something like a catalytic reaction, if you're in green chemistry and you want to convert CO2 into sustainable aviation fuel, in order to simulate that correctly, you need to simulate thousands, if not millions of atoms, and you need to simulate that over very long time scales. And so the thing that I think is extraordinary about AI is you could take just a tiny piece of that simulation, maybe just 100 atoms, and you can run that simulation, which isn't going to tell you very much about chemistry, but it's going to give you rules and insights into physics, and you can learn your algorithm on lots and lots of examples of very small systems. And then that's going to generalize to very big systems when you then go to inference time. And I think that's the crucial observation that really accelerates and is the game changer, because what you're able to do is take something which is too simplistic to be useful in prospective design of materials, but you can learn the rules from that thing. And then you can apply that into the large scale systems that do tell you something useful that would be completely impossible to be tractable with traditional approaches, but work with AI. So that generalization capacity of these new AI algorithms, I think, has been just so game changing. And I think Tim has been really leading the way here with some of the ORB models. The thing that really just blew my mind was training on small inorganic crystals, like 20 atom systems, to then simulate a protein through out of the box generalization. That's telling you that we're learning something really fundamental at that small scale, which I don't think anyone had ever expected.

Tim Duignan: (34:47)

Yeah. That's a great point. There might be like kind of a finite amount of complexity that you have to capture. And then once you've captured that, you can kind of do a huge amount of other things. There's only so much data you may need for some problems, I think.

Nathan Labenz: (35:00)

Yeah. Hey, we'll continue our interview in a moment after a word from our sponsors.

The Cognitive Revolution is brought to you by Shopify. I've known Shopify as the world's leading e-commerce platform for years, but it was only recently when I started a project with my friends at Quikly that I began to appreciate just how dominant Shopify really is. Quikly is an urgency marketing platform that's been running innovative time-limited marketing activations for major brands for years. Now, we're working together to build an AI layer, which among other things, will use generative AI to scale creative aspects of their campaigns. Those campaigns typically involve giving consumers a deadline and they include things like fastest finger wins, photo contests where customers submit pictures of themselves using a product, and even personalizing promotions to individual customers' histories. In all cases, they're constantly scanning for fraud and abuse, and they're also increasingly using AI to both understand and engage with their customers. As we built the AI layer, our friends at Quikly, who are Shopify insiders, are working with the Shopify ecosystem on a daily basis, and they've told me about the just incredible scale and quality of the data that Shopify handles. Not only is the Shopify platform painstakingly designed for ease of use and to make setting up a store dead simple, but their data pipelines are equally robust. They handle millions of transactions per minute at peak times, providing real-time inventory, customer, and order data to the apps that are built on top of Shopify. So whether you're a first-time founder setting up your first online store or you're an established brand that needs to handle transactions at huge scale, there's really no reason to look anywhere else. You should know that by now, but what you probably don't know is that Shopify is also an AI-powered business. From the shopify magic AI assistant that helps you set up your store in natural language to AI-powered customer service and inventory management tools, Shopify is bringing AI to every aspect of running a business. And of course, Shopify's APIs make it super easy to integrate your own AI tools as well. So if you're thinking about starting a business or if you're looking to switch from your current e-commerce platform, you should absolutely check out Shopify. You can get started with a $1 per month trial at shopify.com/cognitive. That's all lowercase, shopify.com/cognitive.

Nathan Labenz: (37:02)

Okay. So you mentioned these ORB models. I was doing my homework, of course, preparing for this, and I understand that there's sort of a constellation of models, but I need your help to understand the hierarchy of how these things relate to each other. So I gather from the website, there's a model called Linus that's kind of the internal, most foundational foundation model. And then those are getting kind of fine-tuned for different purposes. Maybe it would actually be most useful just to talk about like some of the tasks. We've alluded to one, which is like simulate a system through time. And you alluded earlier to like same architecture as the weather simulation models. Those tasks seem to line up, right? Given some initial weather conditions, propagate that through time. Given some initial setup of atoms, propagate that through time. But I've also heard other task types described. For example, can you come up with a material that will have these properties? And that sounds like a very different sort of input output. So maybe just give us a sense for what task types we're trying to tackle here.

Jonathan Godwin: (38:10)

Yeah. So here's where Tim's saying everything is Langevin dynamics becomes really important. We start off with a generative model of materials. So what we mean by that is, it's like a generative model of a protein or a generative model of an image. That's probably the right way to think about it rather than a large language model. And these are what are known as diffusion models or score-based models. And I was about to give you a description of the physical interpretation of these, but Tim will get the maths more correct than me. So maybe Tim, you can explain how these things are actually all just physics all along.

Tim Duignan: (38:50)

Yeah. So the interesting thing about the diffusion models is that if you look into the inspiration, it comes from this non-equilibrium statistical mechanics. It comes from these kind of trajectory generating molecular simulation algorithms, Langevin dynamics designed to generate trajectories of atoms moving through time. So there's a deep connection between this task of generating new molecules and making these simulations of things through time where the mapping is the score, which is this vector field that the atoms move on in a diffusion model that you generate the atoms with, is also basically analogous to the force field, which is the thing that pushes all the atoms in a molecular simulation. And you can actually show—there's a lovely paper from Cecilia Clementi showing that if you train a diffusion model on one of these movies, on one of these simulations, this time evolution, the score is mathematically identical to the force field. So there's deep mathematical connections between these things, which is very exciting. And so the generative model, the way to think about it is it's doing essentially a molecular simulation. You can change the atom types as well. It's not just the positions of the atoms which you allow to move. You can also change the atom types as well. So there's actually deep connections between the molecular simulations and the diffusion models.

Jonathan Godwin: (40:05)

Yeah, it's an extraordinary thing. It allows us to do this pre-training fine-tuning step. So the way I think about it is these diffusion models are learning a simplified physics. The great thing about the diffusion model is that you get to sample infinitely many points, almost infinite training data in a diffusion model. Because the way you train a diffusion model, in the case of an image, you take an image and then you gradually corrupt it with noise. And then you predict—you can do your reverse step where you start with noise, and then you predict how to denoise that image. And that denoising process is the generative process. But you can sample infinitely many noise corruptions. And so you've got an infinite amount of training data that you can train on. So you've got really high data density. You've got infinite samples of your force field. And so that is a great way for doing pre-training. For pre-training, you need a lot of data. So the way I think about our generative model is that it does the generative side. So we get to generate materials according to functional properties of that material, but it's also a great way for us to learn a simplified physics. You've got infinite data for a simplified physics, and that is a great pre-training step for us to learn the real physics in fine-tuning when we're learning the real forces computed through quantum calculations. So the Linus is the foundational, foundational model, learning a simplified physics, which we then adjust to give the correct physics that we can use in real simulations in that fine-tuning step. And that fine-tuning step gives us ORB, which is basically calculating the interactions that happen at a quantum level between atoms. And once you've got that right, then you've basically got physics right. You can then move on to simulating whatever you like. You can simulate nuclear reactions. You can simulate what happens in the body, and you can simulate all the incredible materials that we use every day.

Nathan Labenz: (42:07)

So I get the image one. Let me just stay on this for a minute because I want to understand this better than I do. I've given a very similar explanation to many people many times as you just gave about the image version of this where it's like, okay, I've got an image. I can add noise to it. I have a pretty good intuition for what that means. And I also have a pretty good intuition for how if I just keep adding a bit of noise over and over again, eventually I get to something that, for all intents and purposes, is just noise. And then I can imagine reversing that and training to predict—the way I always phrase the question is, what would this image look like if it were a little less noisy? Then you also have the steered version of that, which is what would this look like if it were a little less noisy and a little bit more like this text prompt that was aligned to image space via CLIP or whatever. Now, in the context of proteins or peptides, I've learned, but not super well understood, that there are custom noising processes because it's just more complicated. I guess if you add pure noise, you end up with something like basically incoherent and it doesn't really work in the same way as you can just sort of add kind of pure noise to an image. And then there's also probably issues with discreteness where you're talking about atom type changing and that seems like not something that is super differentiable. And then there's also this aligning question too, right? Of like, how do these sort of low level detailed configurations of atoms translate to qualitative macroscopic properties that we care about? So, if that was the 101, give me the 201 on what noising means in this context and how you're aligning low level things to properties we care about.

Jonathan Godwin: (43:57)

So noise is super straightforward in materials. It's very similar to images. You don't have to treat it specially. You take a crystal and then you just jiggle the atom positions around. And you trickle in more and more noise until it becomes totally incoherent. And then there are techniques that you can use for discrete diffusion as well. And they're kind of pioneered in images, and they work pretty well when you transfer them over to materials. So it's actually more similar to images than it is to proteins in some ways. The only thing that is super weird in a crystal structure that you have with a material is that you've got a box and you've got your crystal inside that box. And if an atom moves across one face of that box, it appears in the opposite face again. And so noising means something slightly different because you don't have an opportunity to go all the way off into the distance. You're always going to be sitting inside one of these small boxes. And that has a little bit of challenges to the maths, but in practice, you just treat it like you would an image. So it's pretty straightforward in that sense. When you think about guiding towards property, it actually ends up being very similar. In an image, you will train—I mean, there are a bunch of ways in which people do this, but the way you do sort of guidance—the way you do that is you create an image classifier basically. And that creates an energy function, something that says, is this close to a cat? And as that number goes down, it becomes closer and closer and closer to a cat. We just do the same thing. We will take a crystal structure or material and we'll train a simple supervised learning algorithm to predict a property. And we'll set that property to what we want it to be. And we'll just nudge it towards that property through that differentiable machine learning classifier. So your description is basically entirely the same as what we do in materials. The thing that we have to do differently is we can't use the same machine learning architectures. We may need to make adaptations to our underlying machine learning architecture in order to get really accurate results within material science. That's the only big difference between materials and images. If you were to look at the maths, the maths looks pretty similar.

Nathan Labenz: (46:26)

Sorry, could you say again what the big difference was?

Jonathan Godwin: (46:29)

What's really important—if you have two atoms and they get closer together, then the forces that are acting on those atoms increases. And then if atoms go really far apart, you get no forces applying on them whatsoever, but there might be a sort of Goldilocks zone where they're attracted to each other. And so the distance between two atoms or the vector, if you're thinking of the 3D space, is like a first class citizen in representing an atomic system. It's not a first class citizen in an image because all the distances, if you think of a pixel as an atom, all the distances are the same. So you don't need to represent it in your architecture. Or all tokens are the same distance apart. So you don't need to represent distance in a transformer. But for 3D point clouds, for atomic configurations, distances are first class citizens. You don't get to predict the property unless you have a really good high fidelity representation of distances. And so that means that you have a different type of modality as an input to your machine learning model. You need to have distances of which there are a quadratic number, and you need to have the atoms themselves. So that gives you a very different type of input modality. And so you need to have a different type of machine learning architecture called a message passing neural network, which is basically just like a transformer, but a transformer that takes distance inputs as well. And that is what we use. That's the major difference. If you were to be training, look at the code for our models, and you just look at the code for a vision transformer or language model, that's going to be where the major differences are.

Nathan Labenz: (48:37)

And I would love to have a little bit better intuition for how we make those leaps from atom type to atom type.

Jonathan Godwin: (48:44)

Yeah. So you basically treat categorical predictions—so the atomic species from 0 to 117—as a continuous function. So instead of having a cross entropy loss on it, you treat it as a continuous target, and then you cast that back into a categorical one. So this is a technique—I think it's called binary analog bits. It came out like 2022 as a way of doing discrete diffusion. You basically just cast it as something that has a gradient in it as a kind of softening of that discrete distribution. And then it's a pretty elegant and pretty simple technique that works pretty well. Yeah, it's a Hinton paper, you know it must be good, right?

Nathan Labenz: (49:33)

Okay. So can we go back to tasks then and say with all that in mind, when I do an image diffusion model, I might show up with just a text prompt and say, make me a picture of a scientist in a material science lab. Or I might say, here is a picture and I want you to make it more like a dinosaur working in a material science lab. What do I show up to the different ORB models with and what do I then get back?

Jonathan Godwin: (50:05)

Yes, for Linus, you'll show up with something like, I want to have a band gap of X, give me 10 materials that have this band gap, and it will generate a material that has that property. Very similar to an image generative model. You start off with a description of the image, and you'll generate an image that fits that description. It's exactly the same. That's what Linus does—our foundational foundation model. And then all the ORB models that we've released, the ones that are publicly available, have a different task, and that's the simulation task. So that is, I've got a system, but I want to know what's going on. I want to know from a first principles perspective, how is this reaction going to be taking place?

Tim Duignan: (50:51)

So the main thing they're doing is predicting forces from coordinates. So you give them the positions of all of the atoms and the atom types, and they'll print out the forces and the energies. That is essentially all they're doing, but that actually unlocks a huge amount of other things you can do, right? Because people have been running these simulations for decades and developed all of this theory and code and know-how about how to use this ability to run simulations with these forces. Then we can leverage that to compute a whole host of other properties. So you can compute things like diffusivity, so how quickly atoms will move, their chemical equilibria, so how strongly they stick to each other and stick to other atoms. Essentially, all of the really fundamental stuff you want to know about materials, you can then extract from these molecular simulations, which is a really exciting thing. So there's then kind of two ways you could go about generating new materials that have the properties you want. One is you could try and just directly simulate a ton of them and see if they have the properties you want. And then you can try and bypass that with a direct generative model. And you can combine them in different ways, right? So coming up with hypotheses with the generative model and then feeding them into the simulation to check that you do see what you want with these more exact simulations.

Nathan Labenz: (52:03)

Yeah. I kind of just want to say keep going there because the sort of outer loop is really interesting. I mean, we talked about that in the context of your work with the solutions, and it was like calculate all the forces, apply a time step increment, everything moves a little bit, recalculate the forces. Next thing you know, you're seeing crystallization popping up out of nowhere. I'd love to, yeah, kind of keep zooming out on the tasks and talk about the scaffolding that you're building around them.

Tim Duignan: (52:34)

So in terms of molecular simulation, there's really three key things you can do with it, which is exciting. One is I think of as a computational microscope. So the idea is you just directly see what's happening. So you can make a movie and you see what the atoms are doing. This is a very powerful and useful tool because in particular, it gives us access to a temporal and spatial scale we haven't had access to previously. And that's particularly the nanoscale—nanometers, nanoseconds, kind of a billionth of a meter, billionth of a second. This is a really important scale because a ton of interesting stuff happens there. All sorts of chemistry, chemical bonds are formed and broken at that kind of timescale and spatial scale. And so we'd really like to be able to see what happens. It's very hard to do it experimentally because these things are much smaller. They're atom sized. So you need something similar size to kind of observe them experimentally, but we can potentially get it computationally, which is exciting. And, Amelie, your previous guest, talked about this in the protein folding context, right, where the statics problem is maybe being solved. You can now predict the static structure of all these proteins. What we're really interested in now is how do they move, what are their dynamics, and particularly, these very short timescales, which we can't get to experimentally below a microsecond. And that's where these techniques are really useful. This idea of a microscope is really powerful, I think, because it's revolutionized science many times. Right? When Leeuwenhoek built the first microscope, everything he looked at, he made new discoveries. He just looked at some pond water. Found bacteria. He looked at his blood, he found red blood cells. Just discovery after discovery. And so the hope is that if you can simulate a ton of things, you start to make important and new discoveries and find new things out just by looking at them, which is very exciting. The second one you can do is, I think of as in silico experiments. So you don't now have to go and order the chemicals and synthesize the new molecule and do all this work to try and then see it in action and then use microscopy to try and understand what's happening. You can just change the atom type in your input file and rerun the simulation. So you could do simulations much more quickly, hopefully, than you could do them experimentally. This is a while off, but eventually you could think about doing high throughput things with this. So trying thousands of simulations and just trying to brute force find out which material works best. The third thing you could do, and this is actually long term the most exciting one, is actually use those simulations to extract data from them to then use for larger scale models. And I think of this kind of as providing a bedrock for physical simulation. For many years, we've simulated all sorts of physical systems—the climate, rockets, whatever. You simulate these things with all sorts of modeling approaches, but these all need to be parameterized. They need to have numbers plugged into them. And it's often very hard to work out what those numbers should be. You have to do experiments. Very hard to do those often. Whereas if you could use lower scale simulation to extract that information, you could plug it into these models, which would be very useful. And then the other thing you could do is you could generate training datasets this way, which you could then go and train your generative models on, which would be great as well. And then there's also this coarse graining thing I've mentioned, which is what I was doing with the crystallization. And this is a statistically mechanically rigorous, well-defined procedure where you can show that you can ignore certain aspects of the system and keep other ones. You could ignore the solvent, for instance. So one day, you could think about simulating how proteins fold with time. And in fact, people are already starting to do this using some of the approaches that Amelie was talking about, training on all atom molecular dynamic simulations to then enable you to look at how proteins fold dynamically, which is very exciting, and that will come. And it's very similar in material science. You want to know both the static structures these things are forming, but also the dynamics, how the chemical reactions happen, what are the transition states, all things that are very hard to get at experimentally now.

Jonathan Godwin: (56:21)

Maybe we can say how all these things link together. Really, how do we go from idea to material using these technologies? That'd be helpful.

Nathan Labenz: (56:31)

Yeah, please.

Jonathan Godwin: (56:33)

These technologies are used in a bunch of different ways. We pay a lot of attention to making them really useful in the prospect of designing materials. You often start off with a functional property that you care about. You'll decompose a chemistry problem into a subset of things that you care about in the material. So maybe you're saying something like, I want to create a metal-organic framework type of material. It's a big, sponge-like material. And you say, I want to have a very big pore size. The bubble in my sponge needs to be really big because it needs to contain a lot of stuff I'm going to absorb. So I need a large pore size, and I might want to attach little tendrils, little ligands to this metal-organic framework. And so it needs to have lots of active sites for me to attach these little ligands, which are going to act like little tendrils that pick up the things that I want to absorb from my mixture that I'm going to pass this whole thing through.

So you'll use a generative model to say, okay, generate me a material that has a large pore size and has lots of sites for me to attach ligands to. And that's what the generative model does. It will give you maybe 10, 15, 20 different candidate materials that have some of these properties. But making 20 of these materials is still going to take you quite a long time, and there might be a bunch of things that are really difficult to use as generative prompts but are really important, and we know we can access through simulation. Maybe we want to understand, what are the kinetics going to be? How fast is this absorption thing going to take place? Am I going to block my entire pore space with these little tendrils and say, I won't be able to get access to anything I want to catch? All these sorts of questions, which are really hard to put down in a prompt, really hard to condition against, but simulation can answer really, really well.

So then we'll go through quite an in-depth process of using our AI-accelerated simulation tools to answer these questions at unprecedented levels of detail and accuracy. By the time we get to making a decision about what to make, we've answered 90% of the questions that we need to in order to feel confident that we're going to have that sort of material. And so we're using the generative side to have that creativity. And then we use the AI side for qualification and for giving great insight into what's going on. That stimulates more creativity a lot of the time because you see something unusual. Tim is always coming up with, I hadn't expected this thing to happen. I saw this simulation, and this was not at all what the literature had said, but I actually think this agrees better with the experimental data. You see these new things, and it stimulates creativity, stimulates new ideas for new types of materials. So you might go through that process two or three times, and then you'll end up with two or three things that you've really fully qualified that are probably slightly outside of the scope of what you'd come up with by yourself. The creativity of the AI and the generative processes put you in a different direction. And then when you go into the lab at that point, your success rate is just way higher. You're pretty confident it's going to have those sorts of properties. You're doing something more novel than you would otherwise do. And that novelty point is just as important as the success rate, because you're going into new areas of chemical space. And that's often where the really exciting materials and chemistries are. That's the kind of process that we use, and we use these different tools and functionalities at different stages of the materials design R&D function at Orbital.

Tim Duignan: (1:00:14)

Going back to the very beginning of that, when you say you want something with a large pore size...

Jonathan Godwin: (1:00:18)

Yeah.

Tim Duignan: (1:00:19)

And lots of active sites. Is that something you're saying in natural language, or is there a well-defined property space that you are specifying that in?

Jonathan Godwin: (1:00:30)

Those are the things that you could have a well-defined property space on. I think we've generally found that language in some ways is not the best conditioner for a generative model of materials, because you're often looking for quite specific scalar values rather than a textual description. Text is great for describing an image because you don't have a scalar value with which to condition on. But if you want a specific number for your pore size, it's better to just give it as a number than it is to give it as a free text box.

Nathan Labenz: (1:01:07)

Yeah, because a lot of things you're trying to hit a sweet spot, right? So you don't want yeah. And it can be quite quantitative. You want the right but not too much Goldilocks. Right?

Tim Duignan: (1:01:18)

That makes sense. And maybe one more just detailed question on the models themselves, and then we'll get into the application that is kind of the new headline. How big are these things and what is the sort of compute requirements that go into them? Are we talking, Biden executive order qualifying 10^26 FLOPs if we wanted to call this more of a bio type of thing? Or yeah, where are we in terms of just resources into these models?

Jonathan Godwin: (1:01:47)

I think your mental model should be like an image model, like a CNN-based diffusion model, right? Like these UNet models, which I guess most image generation models are still probably UNets. So I think you should think of it like that. Similar levels of data, similar levels of compute. So a lot smaller than the largest language model. That's one of the reasons why we can deliver so much in world-leading models and lab and developing materials on less than $40 million raised. That's obviously still a huge amount of money, but we wouldn't be able to train a large language model for that much money.

Nathan Labenz: (1:02:28)

Yeah. I mean, the exciting thing is to think that we are at that kind of GPT-1 stage. If you really scaled these, I mean, what you could do with them, particularly on the training data side, right? If you threw some of these huge clusters at generating really high quality training datasets, there's probably a lot of room to improve these things.

Tim Duignan: (1:02:46)

Yeah. Okay. I remember we had two guys, Jonathan Frankel and Avi from Mosaic, to talk about their, among other things, efficiency-driving work. They took stable diffusion training under $50,000 last I heard. So I don't know if that's exactly where you're at, but it just as a general order of magnitude for the audience, those are certainly way less than your trillion dollar data centers.

So with that, let's talk about this paper that you graciously shared a preview of with me where you are modeling no less than the potassium ion channel that sits in the membranes of our cells and lets potassium ions in and out in an ordered fashion. So maybe again, we could start with a I think people will probably be familiar with ion channels and cells, but why does this matter so much? It seems like it had been something for you that was a long-time dream to try to tackle a challenge like this. I think people would benefit from hearing why this was such a prominent mountain in your mind that you wanted to climb.

Nathan Labenz: (1:03:52)

Yeah. So it's always been a problem that's fascinated me, as you say. And the basic idea is that the body is really an electrical device. We don't think of it this way, but it is. Just like a computer uses electrons and metal to carry electrical currents around, but the body uses ions. Things like sodium chloride. Sodium is positive, chloride is negative. Potassium as well is critically important. And the way it carries these signals is it creates cells which have a high concentration of ions, potassium ions in this case. And then when it wants to send an electrical signal, it will open up these things called potassium ion channels, and the potassium flows down those and that creates a change in voltage, which then triggers follow-on processes. So it's really operating kind of in a way that you can think of it as an electrical device, and it's how just a huge number of the body's functions work. It's how your muscles contract. It's how your heart beats. All of the neurons firing in your brain, carried by these electrical signals, carried by the flow of these potassium ions in and out of cells. So it's incredibly important.

And when Orb, this model, first dropped, I wasn't working for Orbital, and I saw it, and I thought, wow, this looks really exciting. Really good accuracy. Very, very fast was the exciting thing. And I started to try it out for many different things, and it was doing a fantastic job, always giving me nice stable results. I simulated many electrolyte solutions, which is what we talked about last time, which is what I was very interested in. But the fundamental reason I had always been interested in electrolyte solutions is, apart from the fact that they're very important for climate change, they're also central to biology, right, to this electrical signaling function. And the real key challenge there had been these potassium ion channels, which no one has really been able to fully understand, unfortunately, using experimental techniques or traditional computation. And in fact, we don't even know some of the most basic questions about it. So down these channels, is it just potassium ions flowing, or is there water going down it as well? That's an unresolved question. There are papers coming out on both sides in top journals every few months. And it's kind of a what I think of as a keystone problem or a kind of problem that demonstrates a fundamental lack of some capability we're missing. And so if a new tool came along that could solve it, then I think it would be really important. And so it's always been the dream to try and use the things I'm working on to address it.

And because Orb was doing so well, I started to think, maybe I'm thinking about this wrong. Maybe this is the universal force field, the thing that can simulate anything that computational chemists have been dreaming about for decades. And I thought maybe this thing is finally here or some form of it. And I thought, well, if it was, what would I do? This is the problem that I would go and try and solve. So I set up the simulation, and at the time, I literally thought this is such a silly idea. This is just not going to work. It's going to crash straight away. The models I had been building myself previously were quite unstable as soon as you got out of distribution. So I thought it's going to crash into unphysical things, but in fact, it didn't. It started running stably. I started to see interesting things, and it was just so exciting. I literally started jumping up and down. It was very cool. And the main thing is that the training data looks nothing like these things. It's small crystals, highly periodic, repetitive structures. So it's never seen anything really like this. So the fact that it even just gave something reasonable and was showing interesting things was really exciting.

Tim Duignan: (1:07:14)

Eureka moments. Yeah. Increasingly coming from AI systems. Multiple follow-ups there, I guess. One is, to what do we attribute the generalization from the small crystal training dataset to this quite wonky and in the picture, there are papers and people can just Google a picture of the potassium ion channel to at least get a rough approximation of what the thing looks like. You've got a cell membrane and then there's this sort of wonky thing that has certain symmetry to it that kind of protrudes through both ends of the cell membrane, and through that channel, ions can flow. That's way different from a like, there's nothing crystalline about it for starters. Is this like a divine benevolence sort of thing? Or do we have an explanation for why it would generalize so well?

Nathan Labenz: (1:08:08)

Yeah. I mean, nodding my head at that, I don't know. As I said, I didn't expect it to work. So my general philosophy is I started seeing I don't trust a lot of AI things at first because they can hallucinate and do things, but I've tried to change my philosophy because I think you can go too far in the wrong direction, right, of not trusting them enough. So I think sometimes it's worth trying and testing to see these things. And so it was definitely surprising. I wouldn't have predicted it. If I had to give an answer, I think I would say something about I think it's what Johnny was saying earlier. There's only a fundamentally small, some finite amount of information about molecular interactions that you need to know. There's things like the dipole moment of each atom and how those interactions look, how they spatially decay. And those can change in different circumstances, but there's just a finite set of information that you need. And maybe the crystal structures are enough to get enough of that information out that you can then generalize to simulate a much broader space of materials. I don't know. Do you think that's right, Johnny?

Jonathan Godwin: (1:09:11)

Yeah. I think that's right. I think the thing that's remarkable and continues to amaze me though is the effects of scale. We take scale, especially data scale, engineering excellence incredibly seriously at Orbital. As part of our pedigree as an organization, I think without casting too much shade on the academic community, I think we're bringing that engineering, AI engineering into this discipline, and that has led to extraordinary results in natural language processing, images. And to a certain extent, I think it's part of the result of that, that we're really bringing the best out of these very flexible networks now to train them. And just like you've seen these unbelievable results in so many other areas, applying the same sort of techniques and abilities to materials has given something truly extraordinary. But I really agree with Tim that all the information is there. There's a limited amount. You learn interactions between different types of atoms. You've got these interactions that are the same whether they happen in a protein or they happen in a small crystal, as long as they're there in your crystal structure training data. Then if you've got a network that learns very well and very efficiently, then it should extrapolate really well. So I think it's extraordinary, but half of the things I've seen in AI over the past few years, you kind of learn to expect these things.

Tim Duignan: (1:10:40)

Yeah. Expect the unexpected. Expect to be surprised. Expect to be amazed. Definitely unavoidable these days. What's at stake with this? Is this more like a sort of AlphaGo moment where it's like, holy shit, but it's not going to change the economy all that much in the immediate term, or are there things that get unlocked very practically as a result of a better understanding? And I guess also, what did you learn about the potassium ion channel? How has our state of knowledge advanced due to this?

Nathan Labenz: (1:11:13)

Yes. So I mean, the first thing I need to caveat it with is that this hasn't been peer reviewed yet. I've sent it to some potassium ion channel experts who say it all looks very interesting and plausible and consistent with what we know experimentally. But ideally, we'd like to see some direct experimental confirmation of the things that we saw. But the exciting things that I saw were that water molecules did enter what we call the selectivity filter, the narrow part of this channel. And that isn't normally seen in classical simulations, in the classical approach to simulating these things where they tend to think that the potassium is going through alone. But there's some, although that's mixed, experimental evidence that water is going through, so that was very exciting. It looks like something different there. And in particular, what we saw was a particular hydrogen bond where this little hydroxyl group on the protein was reaching out and kind of grabbing the water molecules and pulling it into the channel, which was very cool. No one's seen that before as far as I'm aware, and it just makes a lot of sense. Okay. There would be some new kind of qualitative phenomena which the other methods don't pick up that explains why they didn't see water going in that we do. And we know if you delete that hydroxyl group, which they've done through mutation studies, the conductivity drops by almost an order of magnitude. So it really is doing something important experimentally. No one's really known what. This seems like a pretty likely candidate to me.

So although it still needs to be confirmed, I'm very optimistic that this now explains this long-standing dispute in the field about what might be missing there. And it's something we can go out and experimentally try and validate now, which is exciting. In terms of why this matters economically and these kind of things, well, this is very important. Actually, a lot of diseases are caused by mutations in the potassium ion channel. All sorts of heart diseases, long QT syndrome, heart palpitations run in my family. If your potassium ion channels aren't working properly, it will stop these electrical signals flowing in the right rhythm and it'll impact the beating of your heart. And so if we want to understand how do we fix those mutations, we may need to understand first how does it work in the non-mutated form, and then we could simulate those mutations and see how they impair the function. And then even maybe one day start thinking about designing drugs to restore that function. And that sounds a little sci-fi, but this is actually already being done for chloride ion channels. Vertex has small drug molecules that can restore the function of chloride ion channels. So I think there are definitely biological applications for these kind of things in terms of real-world impact.

Tim Duignan: (1:13:48)

Cool. Just to make sure I understood the piece correctly about the hydroxyl group, previously observed was that if you make a cell that has a mutation such that that group is not there, then the channel works order of magnitude less.

Nathan Labenz: (1:14:07)

Exactly.

Tim Duignan: (1:14:07)

Yeah. And what you saw that was new is presumably without feeding any of that kind of background information into it, you saw a mechanism that made you say, well, that's probably why that's happening.

Nathan Labenz: (1:14:21)

Yeah. I mean, if I had to bet, I would think it definitely is. It seems very plausible. It was just doing something new that made a lot of sense that I haven't seen reported in the literature anywhere else. So at the very least, it's a very good hypothesis. And if I had to bet, I think it'll be confirmed.

Tim Duignan: (1:14:38)

So how far do you think this goes in terms of and there could be a really obvious hard limit that relates to sort of the structure of the models that have been trained or, and then maybe there's more like a conceptual answer, but we've got these very small crystals training model that lo and behold can simulate something as complicated as the potassium ion channel. It was interesting to read how you sort of fixed essentially a sort of superstructure. You've got this whole cell, right? This whole thing is in a cell and that's in a body and that's in a world, but you've got to draw a box around somewhere to bound your simulation. So you basically said, beyond a certain box, we're going to kind of treat everything outside of that as fixed in place. And so if I understand correctly, there are some more atoms that are represented in the simulation, but they're just not allowed to move in order to kind of focus the simulation in the zone of interest.

Nathan Labenz: (1:15:39)

Yeah. Exactly. So if you go back to early ways we did simulation of these things, they would use these kind of tricks. And so I thought, you know, I could use some of them here with this new technique. What you do now in a modern simulation, and we will do this with Orb now if we could scale the compute. At the time I was doing these, I only had one V100, right? So very restricted in my computational resources. And the basic idea is you want to simulate this, what's called the selectivity filter, this narrow part where everything interesting happens. But you actually need the surrounding atoms because they help stabilize that central part, right, where the interesting stuff happens. And then you have a problem because now you need the outer atoms around those. And so you have a kind of infinite regress and you end up needing a huge system. The way we handle this normally is a thing called periodic boundary conditions, which is what Johnny was talking about earlier, where if one atom leaves the edge, it comes back in the other side. And so you run very big simulations is what people do now with the classical techniques. I just wanted to have a quick look to see is there anything interesting going on with the selectivity filter? So I had to kind of cut that infinite regress off at some point. So I did it about 20 angstroms away. So it's, I don't know, four or five layers of atoms out. And I just froze that outer layer to give it a kind of thing that would keep it nice and stable for long enough to kind of see the interesting dynamics in the selectivity filter. So it's a little bit of a hack. We'll fix it up eventually and do the full simulation of the entire system. The membrane is the other thing. Because this potassium ion channel sits in the cell membrane, which is these fatty layers, you actually need a very big system to simulate the whole thing. And that should be possible with bigger GPUs and a bit more time.

Tim Duignan: (1:17:16)

Yeah. It's insane that this was done on one V100, you said?

Nathan Labenz: (1:17:21)

Yeah.

Tim Duignan: (1:17:23)

Yeah. That's wild. So is it just a matter of scaling up your inference infrastructure? I mean, the model, I presume, has some sort of structural limits too. Right? You couldn't take the Orb model and simulate a whole cell in it if you just had enough inference compute. Right? What sort of constraints are we going to hit as we try to do bigger and bigger systems?

Nathan Labenz: (1:17:44)

Yeah. It's very interesting. I don't know. We'll have to wait and see. I did have a go at simulating the larger one. It got a little unstable, but it's very tricky setting these simulations up, obviously. Need to get the cell membrane has to be the right thickness and all of these things. So there's a lot to do there to work out how to scale these up to larger systems. I mean, the dream is that you can just continually scale them. It's an open question as to whether the current architectures we're using can do that. It's possible you may need kind of longer-range models that can treat these longer-range, particularly electrostatics. In these systems, you get charges which have very long-range interactions because they decay as 1 over r, which is the same as gravity. So they can go to the moon. So you can have these very long-range effects. So it's hard to say what will happen when we go to very large systems, but it'll be interesting to find out.

Tim Duignan: (1:18:38)

But there's not an equivalent of a context window or some sort of hard limit built into the current models that...

Nathan Labenz: (1:18:46)

Yeah. You would hit the memory limits of the GPU if you tried to scale it too much, although that's not really our bottleneck at the moment. We can do huge ones. The problem is it gets a bit slow when you do huge systems. Right? So then you would have to start scaling to many GPUs. People have done this with neural networks, simulated a whole HIV capsid, and it looked stable, not with Orb, but with another force field. So you can simulate huge systems with it. People have done it. It looks pretty stable. It is hard to validate it because the time scales are quite short, right, because it's so computationally expensive. So that's really the problem at the moment is how to get these longer time scales. And I think there, this coarse graining idea will be key. We will run all-atom simulations, and this is what I did for the salt crystallization. But then you say, we're going to ignore the solvent molecules and train a new model, which just predicts what's called, what technically is a free energy now. But it does the same thing as an all-atom simulation, but now you're just simulating a subset of the system, just the pieces you care about, and that will get us to these very large scales. So I'm optimistic that we can start to think about simulating whole cell type stuff with this approach. It will just involve some hard work. But I think the theory is all there. The tools are all there. It's just a matter of really investing in it.

Tim Duignan: (1:19:57)

And there's nothing about the current models. You know, obviously, large language models have just a finite number of tokens that they can handle at any given time. Beyond that, they just can't do it structurally. Is there I guess, there's not a similar hard limit here to a certain number of atoms or whatever?

Jonathan Godwin: (1:20:15)

So what happens in a language model is that when you have a finite size of your context window, that's because you have this positional encoding that is a feature of your input. And so during training, because you only can train from a memory perspective at training time, a certain context window length. Each position in that context window has a specific token, a specific feature associated with it. So at test time, you can't generalize much further than what you've trained on. That's stuff like Mamba or these state space models, where they don't have this explicit positional encoding. That means that you can have kind of infinite context window. And I think similarly, when you think about the sorts of models that we use, we don't have any explicit positional encoding. That means that we can really extend just as far as the compute resources can take us and basically linearly increase the size of your simulation by the number of chips that you can add on to your simulation when you think about an atomistic model. And so really, sky's the limit there. You can make it as big as you possibly want as long as you've got the chips. We don't have any fundamental model architecture challenges like you would do with a transformer-based language model.

Nathan Labenz: (1:21:37)

And so we're exploiting the nature of molecular interactions that they tend to be local, right, which is a nice property that they have, which enables you to kind of do this.

Tim Duignan: (1:21:47)

So I have to ask, when do we get our room temperature superconductor?

Jonathan Godwin: (1:21:52)

It'll happen. Give us at least five years, but it may happen. Before that, we're going to be solving carbon removal and scaling AGI energy efficiently and cheaply. So we've got some things we're solving first, but once we've done those, we'll think about the room temperature superconductor.

Nathan Labenz: (1:22:11)

Yeah. The problem with superconductivity is what we call highly correlated electron systems. So it doesn't have this property of very local behaviors. There are quite complex quantum mechanical things going on, which our the current level of theory we're training on, density functional theory, is not really good enough to describe that. But it could be very useful for finding new crystal structures and, you know, working out how to make new superconductors once you know how to induce the kind of electronic properties you want.

Tim Duignan: (1:22:40)

Interesting. Okay. Cool. Well, maybe talk a little bit more about that sort of roadmap between here and the room temperature superconductor. What else is on your to-do list?

Jonathan Godwin: (1:22:52)

Our focus at the moment is really on data center materials. We think that scaling out AGI is one of the most important things that we can do, and that new materials are going to be a really important part of making that both cost efficient and sustainable. As the rack density increases, as power density increases of the chips that we're using, we need better and better materials for thermal management, better materials for all sorts of different things. We can only imagine that we're going to need better and better chips in order to bring AGI to life. So in that sense, new materials, data centers, new materials for AGI is our focus. We also want it to be sustainable, though. And so decarbonization is our other focus as well. And you'll see coming forward a pipeline of materials and products in this space. So watch this space. We've released information about one of those products, more coming over the course of this year.

Tim Duignan: (1:23:55)

Tell us a little bit more about the one that's already out.

Jonathan Godwin: (1:23:58)

So that's our carbon removal material. This is a material that we've worked on and we'll be piloting in partnership with AWS towards the end of this year. We have designed this material to operate and capture CO2 under the operating conditions of a data center. It allows us to utilize the waste heat and waste airflow of a data center to maximize the CO2 capture capacity and ultimately offset some of the scope 2, scope 3 emissions of data center operators. It's perhaps the biggest, largest CapEx build out in history, what we're currently seeing with the hyperscalers and the data centers now. But there's also companies that are very serious about their climate goals, and certainly have the resources to invest in reducing the carbon intensity of their operations. So we see that this is a great opportunity to be deploying really advanced and really effective carbon removal and decarbonization technologies.

Tim Duignan: (1:24:55)

Yeah. I think technologies like this are the only thing between the hyperscalers and a lot of broken net neutral promises, I suspect.

Jonathan Godwin: (1:25:04)

Yeah.

Tim Duignan: (1:25:04)

I don't think they're going to not build the AGI data centers. So something's going to have to give. Is that material, were you describing it earlier when you talked about a spongy, high pore size with a bunch of active sites to grab onto things? Is that the sort of mental model I should have?

Jonathan Godwin: (1:25:22)

I think that's right. We can't say too much about that because we're currently protecting some of that information, but it's certainly a sponge-like material. And there are reactive sites that should selectively interact with CO2 over all the other stuff that's going to be in the air. The things that we care about, like the absorption capacity, the temperatures that the data center is operating under, composition of the air, things like that. We tailor our materials and develop new materials that are specifically designed and can take unique advantage of that operating condition. That's a really hard thing to do because developing a new material for CO2 capture takes a huge amount of time, years and years and years. We're doing that incredibly fast. And so we're really able to maximize the unique opportunities of that deployment situation. It's just one example of the impact of AI in some of our work.

Tim Duignan: (1:26:18)

Do you have to make, you can imagine multiple paradigms for that sort of material. You could imagine a sponge that absorbs a bunch of stuff and then you sort of bury the CO2 with the sponge and you keep bringing more and more absorbent material and it itself becomes sort of a waste product. Or you could imagine you could clean it off somehow and reuse it.

Jonathan Godwin: (1:26:43)

You're going to do the last bit of that.

Tim Duignan: (1:26:44)

Yeah.

Jonathan Godwin: (1:26:45)

You're going to be absorbing and desorbing the CO2, and that's a really important part of the overall costs of doing the CO2 capture. If you can bring down the heat at which you are desorbing that CO2, or tailor that temperature so that you can utilize the waste heat of a data center, then that's going to really massively reduce the energy costs. But the longer that you can reuse that material, the better. I don't think we're going to get to a point where desorption of that CO2 is going to be more energy costly than making another x tons of that material. That making process is super energy intense and is a costly thing. And if that material can then last for 10 years, then that's one mechanism by which you can radically reduce the cost of that CO2 capture.

Nathan Labenz: (1:27:37)

Yeah, the one-off capture is pretty solved. It's really that regeneration problem that's challenging.

Tim Duignan: (1:27:41)

Making stuff in general and just manufacturing a lot of stuff is obviously quite a different business to be in versus coming up with the models that design the materials in the first place. I know you said earlier that you have traditional material scientists on the team and that helps you close the loop. I assume you're licensing the things that you discover? You're not actually building factories and scaling up production, are you?

Jonathan Godwin: (1:28:07)

I think that's right. Whether it be licensing or whether we have suppliers, you know, NVIDIA doesn't have a fab, but they still sell chips. We don't have to build up manufacturing of material to sell materials. But I don't think that we're going to be standing up a massive plant somewhere in Jersey. That's true.

Tim Duignan: (1:28:32)

Do you need to have a lot of physical capital though to even just do the validation experiments? I mean, it would seem like that in and of itself would be a kind of scale game, right?

Jonathan Godwin: (1:28:41)

Yeah. We need more physical capital than a software business does. I think that's right. But we're not going to be raising money for CapEx. Certainly, our lab is cheaper than a lot of the GPUs you'd need to run an AI company. It's not beyond the scope, I think, of what venture capital funded businesses should be able to do.

Nathan Labenz: (1:29:06)

I think it's so important to have that internally, right? In terms of being able to get feedback really quickly from experimentalists about what's working and what's not working and what they need. That's such an important thing that it would just be so much harder to do without.

Tim Duignan: (1:29:21)

Yeah. Cool. This is amazing stuff. So I'm sure you've seen this paper that made the rounds recently from Adept, Toner Rogers from MIT, where I don't think we know what models are being used or what company this research was done at, but basically a sort of behavioral industrial economics study of the application of models in this general space to the practice of material science. The headline results were significantly more materials discovered, 44% increase, resulting in 39% increase in patent filings and 17% increase in downstream product innovation. Interestingly, the AIs as characterized in this paper are doing a lot of the idea generation and then the scientists were more focused on the validation. So overall, greater output from this system, but one of the important caveats was the scientists, 82% of them reported basically lower satisfaction with their work because I guess they felt like they wanted to be the ideas guys and now the AIs are the ideas guys and they are sort of left to be the hands that validate the AI's ideas. I guess you could react to that in multiple ways. I'm interested in how you think those numbers in terms of increased throughput relate to what you expect or aim for. And then also, is there anything, I've got Tim jumping up and down here. So I guess you're in the 18% who are finding the joy in working with the AIs. I wonder if you have any outlook on this. And maybe it doesn't even really matter. I mean, I'm not one who gets too hung up on, oh, my job changed. If it's going to solve climate change, I'll accept a little lower job satisfaction here and there. But I do wonder what you think this implies for future of work, big picture. Are we going to adjust and find our joy in the work again or is this a sign of things to come? So, yeah, just multiple reactions to that paper if you have any.

Nathan Labenz: (1:31:36)

So on the enjoyment front, my take is I'm a physicist and mathematician by training. And I've been trying to get into chemistry because it's so important and I love it, but I don't have that kind of deep knowledge of it. In fact, I didn't actually study it in undergrad or anything. So from my point of view, these tools are incredibly empowering because they give me access to that intuition that we were talking about. Because now I can just run a simulation, see these bonds forming. And so I just look at these simulations, and I'm like, oh, okay. That's what's happening. And I don't need to know anything about potassium ion channels. I don't need to know the literature that that hydroxyl group is known to be important. I don't need to know a deep intuition about what bonds with what. I just run the simulations and see it. So I have sympathy with the people who spent their careers building up this intuition, and now something else is coming in and can do it as well. So, yeah, I do think that's a sad thing. But I think people, I think there will still be things people can be useful for. And I think the satisfaction of getting materials out there in the world that are having an impact much quicker will more than offset that spark of, oh, I solved the problem myself. You can do crosswords for that, whatever. But this is too important to think about the happiness of the scientists working on it is my take on that.

Jonathan Godwin: (1:32:51)

It reminds me of the job satisfaction of a bunch of my friends who work in AI research. It's declined significantly because you used to do really creative stuff. And now it's engineering, and there are a limited number of experiments that you yourself can run. You can't go and do your own research project anymore. I've got a great idea. Can't really experiment with it because it's so costly. There's just not enough compute resources to test it fully. So people's jobs have radically changed in AI research over the past two and a half years. For many of those researchers, they've declined despite the fact that their jobs have become more prestigious, they're getting paid more. They're seeing AGI take place in front of them. I think that for a lot of people, that's going to be the case. We're going to get more productive, we'll get richer, but we will have to find other ways in which to feel value from our work and that creativity. These systems are really creative. Creativity is one of the first things that's going to go. It's not going to be manual tasks. Robotics will come quite a lot after the creative industries have been fully accelerated through the use of AI. I'm not so optimistic that people are going to enjoy AI. I'm fully convinced that the only way that people are going to enter the future and have a happy world is through AI, because we need to produce more stuff for less money. And that means that we need a greater acceleration of scientific discovery than we've ever had in the history of humanity. And the only way that we're going to achieve that form of acceleration is through AGI. So we have to have it as fast as possible, deal with the pain over the short term, to get to the glorious future, as I would put it, of a large portion of economic activity being fully automated.

Tim Duignan: (1:34:55)

That's probably a good note to end on. Anything else you guys want to talk about that we haven't touched on?

Nathan Labenz: (1:35:01)

One point was just on the combination of LLMs and neural network potentials. These universal force fields are quite powerful. The point worth making is if you're worried about artificial superintelligence and things, a central challenge is how are these things going to get some kind of experimental validation? Maybe they can design whatever new materials they want, but they need to be able to validate those things. This combination of LLMs with the force fields, which gives them the ground truth, perhaps enables them to do these things internally without needing to go out into the experiments and measure things experimentally. So it's going to be very interesting. I think it's a powerful combination because you get that grounding, the way to check hallucinations and correct hallucinations physically. And we're seeing this with these new O3, these models that can solve these incredible math benchmarks, but it needs to be problems where you can validate them somehow. And so I think these neural network potentials potentially provide that for science, a way of validating hypotheses automatically in the computer rather than having to go out into a lab. So I think they're going to be a powerful combination moving forward. It'll be very interesting to see.

Tim Duignan: (1:36:05)

Do you think we end up with omnimodal models where the AGI is something that with the same weights is talking to me and running molecular simulations?

Jonathan Godwin: (1:36:20)

Certainly you'll have a shared embedding space. I don't know, because I think there's some of these image assurance models where the decoder for the image is trained perhaps separately or feels like a different model in their structure, but they've got the same embedding space. I think we'll certainly have that. I think we'll have omnimodal embeddings. That's where the concept of the thinking happens. I think that's almost inevitable. Integration of the atomistic stuff into the training of a scientific AGI, I think, would be a really cool thing to do. And I imagine someone's already working on that somewhere.

Tim Duignan: (1:37:03)

Yeah. I wouldn't doubt it. It's going to be hard to make sense of it. I do sort of worry that for multiple different reasons, obviously there's been deception discourse recently and deception discourse was quickly followed by reasoning in continuous latent space. I am among those who's like, geez, I kind of liked it when I could read the chain of thought. You're telling me now that the chain of thought is going to be all in latent space? That's going to be a little weird. And yet it does seem like it's a very powerful attractor for multiple reasons. I mean, including the systems will be able to communicate with each other much, much more effectively if they don't have to condense everything down to language tokens to send messages back and forth. So I don't know how we get around that, but it does seem like that vision takes us very quickly to a world where we are sort of just hoping the AIs are doing the right thing for us. I mean, is there an alternative vision aside from that?

Jonathan Godwin: (1:38:04)

Yeah. I mean, someone I saw recently said, you just hope moral realism is true. That it's going to find the same, it's going to do the right thing.

Nathan Labenz: (1:38:14)

Alright. Sobering notes to end on, but inspirational work. I mean, really, really cool stuff. And we'll post links, of course, in the show notes, but some of the images back to the crystallization and also with this potassium ion stuff are really, really impressive. So I appreciate you guys' time today in educating me about this. And I'll give you the official send off. Jonathan Godwin and Tim Duignan of Orbital Materials, thank you both for being part of the Cognitive Revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.