Full Transcript

Transcript

Transcript

Anton Troynikov: (0:00) ML models are kind of weird in the way that they think, quote unquote, about the world, and they might represent something quite differently to what a human would. They might focus on quite different things because they are mechanistically pursuing an objective. They are not trying to encode any meaning. The Wright brothers were wrong about how the Wright Flyer worked. They just had intuitions about it, and they employed those. That's an object lesson. It doesn't really matter if your intuition's wrong if the thing works. And then there's this Cambrian explosion of different types of aircraft because all we had to go on was intuition. We didn't have engineering principles yet, and we didn't even really have the tooling to develop those engineering principles. That's where we're at today, in my opinion, with machine learning. And one of my favorite things about Kurzweil is I went on the website not long ago, and it was like the tenth anniversary of The Singularity is Near. And that just made me laugh.

Nathan Labenz: (0:51) Hello, and welcome to the Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my cohost, Erik Torenberg.

Sponsor: (1:13) The Cognitive Revolution podcast is supported by Omneky. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms with the click of a button. Omneky combines generative AI and real-time advertising data to generate personalized experiences at scale.

Nathan Labenz: (1:33) Anton Troynikov is a cofounder of Chroma, the AI-native open source embedding database. In the midst of an AI gold rush, Anton is selling shovels. Embeddings, for any who don't know, are numerical representations—generally speaking, a single vector, or simply put, a list of numbers that encode an input. That could be text, an image, an audio file, or even a DNA sequence or protein structure into a lower-dimensional latent space. The embedding process is itself learned such that semantically similar inputs cluster together in latent space, and the resulting embeddings can be used to compare inputs, to power retrieval systems, and for additional processing. Embeddings are a key piece of a huge number of AI-based projects right now. By embedding a dataset, whether it's a user's email history or Google Drive documents or even a company's entire knowledge base, developers can enable semantic search at runtime and include the most relevant background information in language model context. This dramatically improves performance by reducing hallucinations and increasing personalization. We talked to Anton about embeddings, the tradeoffs and optimizations involved with working with embeddings at scale, their recent Stable Attribution project, which flexes the power of Chroma by taking a Stable Diffusion-generated image and attempting to determine which of the 5 billion images in the LAION training set had the most influence on its creation, but which also quickly became a flashpoint in the broader debate about how artists and other rights holders should be credited and compensated for their contributions to generative models. We also covered the big picture of recent AI developments. Anton is a deep thinker with a sophisticated conceptual understanding of latent space. I learned a lot from this conversation, and I hope you do too.

Nathan Labenz: (3:32) Anton Troynikov, welcome to the Cognitive Revolution. Glad to be here. So tell me first about vector databases. I think our audience is obviously interested in AI. They most probably have at least a superficial sense of what a similarity score is, what a dot product is. But take me through how that works as a database, and then we'll get into some of the optimizations there as well.

Anton Troynikov: (4:00) Yeah, sounds great. So let's start at the beginning. What a vector database allows you to do is, if you can represent your data as a vector, it makes it possible to perform geometric operations on your data. So if traditional databases have algebraic discrete data structures—like a SQL table you select from where, and it has concrete properties, and it returns exactly that result—with a vector database, data is represented as vectors in high-dimensional space. So you can perform operations that have geometric meaning instead, like distance. You already mentioned similarity scores. You can talk about densities meaningfully. You can fit surfaces to them. You can cluster them in a way you can't in an algebraic or discrete data structure. So I think the reason that they're exciting right now is because we now have powerful models that can transform all kinds of different types of data into vectors, and transform them into vectors in a meaningful way. So traditionally, that's been text, and a lot of these embedding functions have been hand-rolled. But now with the success of transformer models and attention-based models for modeling text, the vectors that you get out from those kinds of models are much more meaningful. They represent structure like a human would impose structure on the resulting vectors. So sentences that talk about similar things end up close together in vector space. And that's kind of what's turned on a lot of experimentation and development with using embeddings as the AI-native way to represent data. And so I'll give you an example. A typical workflow that you see is you want a model to answer questions about specific information. So ChatGPT, for example, is trained on an enormous amount of data. You can ask it questions about anything, but it might hallucinate. It might not have the information that you want. And certainly, it might not have your personal private information that it wasn't trained on because it's not available anywhere else. You still want it to be able to answer questions about it because it's good at general question answering as a task. So what you can do is take the corpus that you want it to answer questions about, and then you embed it. You can embed it using a bunch of open source models. You can also use OpenAI's API. You can just use their API and transform that data into vectors. And so you have the data available as vectors with the actual text associated with that representation. And then what you do is you query it. And you query it by taking a query, embedding the query text, performing a nearest neighbor search or an approximate nearest neighbor search in the vector database, returning relevant documents. And then what you do is you take your question and you take the relevant documents that it found, and then you ask the model specifically to answer the question with the documents that you've put in its context. And then it's much more accurate, and it's answering questions about the thing you actually wanted to answer questions about with data that wasn't available to it at training time. This is a really powerful technique. It pretty much suppresses hallucination. It allows you as an organization to use large language models without fine-tuning them, which is expensive and slow. And it confers a lot of other advantages as well, which maybe we'll get into in time. But that's the basic idea. Vector databases have been around for a while. Pretty much any service that you use that has a recommendation or a similarity thing—so the one that I always think of is Pinterest. It'll show you similar images, and the way that's being done is because they have these images encoded into vectors. They've got a large vector database so that when it wants to find nearest neighbors, it'll look for that and you get visually similar images because their embedding function produces visually similar images for vectors that are close together. But the use cases for these things are exploding now because the embeddings for all kinds of data became available, and it's super cheap. You can just hit OpenAI's API. You don't need to run the model anymore. And people are building all kinds of really cool applications on top of this stuff.

Nathan Labenz: (8:06) Yeah, okay. There's a lot to dig into there and follow up on. For one thing, just to contextualize a little bit, I would imagine a lot of our listeners have experience querying SQL databases. So I would love to get a little bit of a sense for how the DBA life is different as we move into a vector world. I think you spoke to this, and I'll just give a couple additional notes on this as well—there is so much energy around these chained retrieval and language model systems right now. That's really at the heart of these dreams that everybody is dreaming. Everybody sees ChatGPT, and everybody says the same thing that you said, which is, well, man, if this had my data, if this could go read all my emails, if this could remember every text message I've ever sent. And then, obviously, the business use case is even bigger and probably a deeper need because now you've got organizations and all of our Google Docs and our whole knowledge base. I mean, how many companies have rolled out these knowledge bases or wikis? Can anybody even locate the information in there? Generally not, right? So this vector database sits at a really critical point in that value chain. And you've started a company that is a vector database company. So before we go into even more details, let's talk about your company specifically.

Anton Troynikov: (9:37) Absolutely. So I want to say one specific thing here. Chroma uses a vector database as a technology, but the thing Chroma really is, is a platform for embedding knowledge that your machine learning-enabled applications can use. You can think of us as the thing that acts as the storage layer for applications that use large language models in the loop. So we're much more than a vector store. And one of the things that we had to really learn while we were building this product was the affordances of the existing vector stores on the market don't really suit this use case very well at all. And we actually built this product in-house, and we were using it in our own product experiments for a while. Chroma basically started on the principle that you can understand a model's behavior based on how it sees its data. An embedding is a representation—it's a model's representation of its input data. And so all of our early product experimentation was around running algorithms on embeddings, representing them, showing them in different ways, manipulating them in different ways. And we needed something really easy to use for developers. We needed something that would allow us to rapidly experiment, maintain development velocity while still being very performant. And so when it became clear—and OpenAI did a lot of good work here in marketing the power of embeddings to people—and these early use cases showed up with document question answering, it became clear that, wait a minute, there's thousands or hundreds of thousands of developers in the world that are going to need something exactly like this. And we essentially took what we built, packaged it up to make it a lot easier to just get up and running, and then gave it to people, and people really seem to love it. Things have been really exploding over the—I mean, it's kind of crazy to me because it feels like it's been out for a while, but we launched it last Tuesday, when I announced it on Twitter. So it's been a lot since then.

Nathan Labenz: (11:29) Yeah. Time is doing very weird things for me as well. Earlier this week, I realized—it was on Wednesday—that I realized, wait, we just did that on Tuesday? And it was like, oh my god, so many things happen in such a small period of time in the AI world. It is dizzying, to say the least.

Anton Troynikov: (11:50) Yeah. I feel like there's been a few step changes over the last couple of years. I think first of all, GPT-3 was the watershed back in 2020, and then Stable Diffusion, DALL-E, but Stable Diffusion was the next really big one because it allowed people to see that, wait a minute, you don't actually need to own a huge model. I can just run this on my laptop and get results and build things around it. That was a huge watershed moment because it was open source and small enough to run on an M1 Mac. You can do it yourself. Someone was crazy enough to get it to run on an iPhone, which I thought was amazing. And then the next thing that just recently happened was ChatGPT. And technologists will all tell you the same thing. People who have been following closely for a while will tell you the same thing. They're like, oh yeah, the model was already available. It just needed fine-tuning. Nobody did it. It doesn't matter. What matters is it's packaged and given the right affordances so that people can easily understand what it is they can do with it. And then they'll start doing that, and then they'll start experimenting. So I think those are the two big watershed moments that happened in the last while that really make it look like an amazing playground for experimentation right now.

Nathan Labenz: (12:55) So you mentioned a couple things that I just wanted to elaborate on a bit. First, it sounds like you guys are a neutral vector database technology, so I can use any different provider. You mentioned OpenAI. Their pricing, just to give context on exactly how cheap it is, because you said it is cheap: the cost to embed all of the transcripts of every podcast that Joe Rogan has ever done, 100,000 pages of text, is $40 to embed that 100,000 pages. And if you wanted to go to the full web scale dataset of 10 to the 11 pages of text, then you're looking at, he says, only $50 million to embed. That's not nothing, but to embed the entire web scale database, it actually is amazingly cheap. So once you have a document or whatever it is embedded, then you can also store those embeddings. Unlike the other APIs that are generative and doing something new each time, the embedding is kind of a one shot. There's no temperature setting. You're going to get the same thing out every time. So then you can store it, and that's where your platform comes in. You mentioned that there are some shortcomings of the things that were out there. So what shortcomings did you see and how did you address those in the product that you've built?

Anton Troynikov: (14:26) Those are great questions. I think the first shortcoming was just how hard it was to get started with anything out there. We evaluated literally everything else in the market, and it just didn't fit our needs of rapid development, rapid experimentation. It was either too hard to deploy and keep running because it was really designed for heavyweight workflows from day one, or it didn't really provide the kind of price performance point that we needed, or it was just frankly too complex. Chroma, for a long time, was just me and my cofounder Huber. And we couldn't spend an entire full time engineer's worth of work just running the vector store part. We needed something else. And the other thing is we found the abstractions were pretty wrong. So there was a bunch of missing features which we just couldn't figure out why they were missing. We just implemented them. There was a bunch of other things too which kind of made it obvious that the affordances here were wrong. None of the other products on the market are really AI native. That's the way we think about it. And that's because if you have this app that's LLM enabled and it has a store, you need a much more transactional approach. You need good inserts, updates, deletes. You need to have it talk to the embeddings API properly. You need to have it return the results in a way that the model is going to pick up easily and then stuff them into the context window, all of these things. We also saw in parallel the growth of what we think of as application development frameworks for AI LLMs. Our friends at LangChain, for example, where we were a day one integration partner when Chroma launched with them. We saw that the thing that we build has to actually fit what people are doing. It shouldn't fit this abstract idea of vector database. It needs to be an app development platform. It needs to be the storage layer for app development. And then it needs to have the right affordances to think about this so that developers can actually think about it in the right way. And those are kind of the main differences. That said, it's pretty performant. We're making it go even faster pretty soon.

Nathan Labenz: (16:25) Pre AI, what were the main use cases for vector databases that people had built all these other things for?

Anton Troynikov: (16:33) The two big ones, as I mentioned earlier, are recommender systems and semantic search. Prior to embeddings, embeddings are a concept that basically just means take data from one high dimensional space and encode it as a vector in a lower dimensional space. That's all embeddings really means. It's actually not even a deep learning concept. It predates it by a long time. It's an idea in mathematics, which the name is loosely used now, but that's pretty much what it means. And so there were a lot of hand tuned methods to convert words into vectors or convert corpora of documents of text into vectors. And then you would try to hand engineer these embedding functions and then store them in a vector database and then query them and look them up in a similar way to the way I've described. And what people focused on in that era was building these really large scale vector databases intended to work at very large scale. So we're talking about being able to query billions of documents out of the box, being able to search over billions and billions of images out of the box for these web scale huge applications. And at that time, AI just wasn't in the picture. Deep learning really wasn't in the picture.

Nathan Labenz: (17:54) So do you also have a SQL-like aspect to the database where you have a could I do the equivalent of a where clause where I'm only interested in these, so kind of traditional b-tree indexes along with the vector space?

Anton Troynikov: (18:15) That's right. We have the underlying algorithm that we use for the vector index, and the index is the thing that actually stores the vectors and the relationships between them. And then there's a metadata store. The metadata store stores the data in original, more SQL-like form. And we have them talking to each other in such a way that you can take something that filters the query that you perform in this very traditional way, and then turns that into a query onto the vector store. So I can say, find me all the matches to this query, but only from documents that contain this phrase. And you can just do that with Chroma.

Nathan Labenz: (18:50) So one of the things you said about the technology that caught my ear was the phrase approximate nearest neighbor search. Tell me about that. That sounds, and this also starts to preview a little bit the project that you've released and made some noise on the Internet with, which is stable attribution.

Anton Troynikov: (19:12) Exact nearest neighbor search has n squared complexity. Each time you add a new vector, and you want to query it to find out what the nearest neighbors are, you need to perform an n squared calculation. It grows in the number of vectors quadratically, which is actually for certain sizes of data totally fine. Roughly, if you have a couple hundred data points, it's fine. You should just be doing exact nearest neighbor. It's just one matrix multiplication and that's very fast. NumPy calls out to very fast numerical libraries. You can just do a matrix multiplication, keep it all in memory. It's nice and fast. You don't have to think about it. The issue, of course, is quadratic growth is really bad. So by the time you have north of 10,000 data points, that n squared is really starting to hurt you. And this is where approximate nearest neighbor comes in. There are several of these algorithms, but basically all approximate nearest neighbor algorithms are flat in the number of or approximately flat in the number of vectors in the index. So you no longer have this n squared term. Your query times, regardless of how many vectors are stored in the index, are going to be identical. But there's a trade-off. The trade-off is you won't necessarily get all the nearest neighbors every time. You might lose some. And that's called a precision recall trade-off, and it's very similar to other aspects of machine learning in that way. But in practice, in many applications like these, that's a trade-off that you can make, and the trade-off isn't very severe. It's not like you're going to lose half. It's more like you might lose one in every 100 queries.

Nathan Labenz: (20:40) Yeah. Interesting. Okay. So that makes a ton of sense if you're working at a 5 billion vector scale. How do you think about, when you said 10,000, I was thinking, I'm sure I have more emails in my Gmail than that. And so what do, how should I think about these sort of mid scale personal datasets where I probably do want the full thing? I don't want to miss the one thing that I most need. So tell me how in that mid scale, what happens?

Anton Troynikov: (21:14) Yeah. So especially for smaller datasets, you can tune for more recall. You can make it very unlikely that you'll miss anything but still get that good query performance. You can also trade off processing upfront and sort of insert time to insert something with recall and query performance, depending on what your workload looks like, basically. If you have a lot of inserts, if you want it to be fast, it's probably a good idea to tune it to be a little faster so that insertion speed, a little bit less recall. If you have a set of data and you really need to insert it all at once, do all the preprocessing on that side. You'll still get great recall.

Nathan Labenz: (21:49) So the tuning, it's interesting. There's multiple meanings of tuning. You're speaking about a pretty classic database tuning paradigm, which is when exactly do we do stuff, and do we precalculate? When, so I had listened to your conversation with Packy on the Not Boring podcast not too long ago. And actually, as I was listening to that, I thought you were tuning some sort of learnable, some sort of aspect of either the embedding or the way, which dimensions would be kind of primary? So tell us about that side as well.

Anton Troynikov: (22:33) Yeah. So this is the other, this is the, again, as I mentioned, one of the most important parts is you can perform geometric operations, and because of the way that embeddings work, those geometric operations have meaning. So for example, kind of the simplest example of this that I can think of is you can cluster vectors geometrically, but those clusters will have semantic meaning. So if you have a large corpus of text and you look at what the clusters might be in it, and you look at the topics that are in those clusters, that's information that belongs together because the model is trained to keep things that seem together together. Clustering is this basic first approach thing you can do. Another thing you can do, and this is actually great for the question answering application, we've been examining this in some detail lately. You can find a direction in the vector space that represents some human understandable property. And it's actually surprising how often this comes up. But for example, you could be talking about a topic, but you can be talking about it in the context of a question, like who was born in year blah blah blah. And then over here, you could be talking about it like, oh, this king was born in blah blah blah. And so if you find this question answer vector, you can do interesting remappings and bring things closer together. And because it's just a vector space, it's this well understood, fairly straightforward mathematical structure, the way to transform one thing that you want into a different thing that you want, or make it work better or fine tune it to get better results for you is actually very straightforward compared to traditional machine learning. You don't need to do backprop. You just need to estimate some transforms, which is something that we know how to do.

Nathan Labenz: (24:10) That sounds pretty similar in some ways to classifier guidance almost. You're sort of identifying a direction that is the direction from statement to question, essentially. And then you can kind of move around, you can place that vector anywhere in theory and kind of move from different statements to their corresponding questions. Obviously, it's going to be a little messier than that. But

Anton Troynikov: (24:39) There's also all kinds of other things you can do directly by working with this. For example, if you have a system to get user feedback about relevance of the returned results, you can reweight your space by applying a learned transform on top of it to get ever more relevant results from human feedback in a very straightforward way. There's no model retraining that you need to do. Your model is doing what it's best at, which is understanding and recomposing knowledge. And actually, that takes me on an interesting tangent here, which I really do want to indulge briefly. Right now, we're seeing ChatGPT and then GPT-3.5 and GPT-4 soon, which are these large models that know things, and they're trained on very large corpuses of data. They're very expensive to train. They're expensive to fine-tune. We don't really know what knowledge is in them, but they're very good at general purpose tasks. They can do all kinds of tasks efficiently for you with zero-shot learning or multi-stage prompting. And I think those models are going to be around. I think they'll continue to be there because they're such great utilities really. But on the other side, I think what we're going to see is these smaller, leaner models which are actually trained to find and compose knowledge in response to queries rather than store knowledge in their own weights. And we've seen early signs of that. There's a paper called Retro, or there's a system called Retro. It's from DeepMind, sorry, I misspoke initially. They trained a model to do exactly the sort of stuff we're talking about. They trained it instead of storing knowledge in its weights, they trained it in such a way that it has to search for knowledge in this data bank. It can't get knowledge from anywhere else except in this data bank. And I think that's one direction in the future, and we're starting to see the very first stage of that happening right now. And the sort of stuff we're building with Chroma really enables that. So those transforms that you described, those are just like one layer. You're basically just learning one additional vector? For a while, there's been folk wisdom, and it's actually even in the OpenAI cookbooks. There's been this folk wisdom that a linear transform, which is just a matrix multiplication, is enough to transform one embedding space into another embedding space as long as the semantics are broadly the same. And there was recently another paper about this where they demonstrated that, yeah, empirically, models tend to learn the same representations of the same thing. So mapping between them is a straightforward operation. And that means that's very promising for being able to do things like that and integrate it really well with what we're building.

Nathan Labenz: (27:06) Like, Relative Representations? Is that the paper you have in mind?

Anton Troynikov: (27:09) I think so, yes.

Nathan Labenz: (27:11) Yep. That was a fascinating one. You see these kind of, I mean, obviously, it's a higher dimension than projected back down to an even lower dimension for visualization, but you see these kind of shapes where you're like, oh, wow. That's very strikingly the same subject to a rotation or a dilation or whatever. I thought the clarity that they achieved in demonstrating that was pretty cool.

Anton Troynikov: (27:34) It was great empirical work. Honestly, I think more machine learning research needs to take that empirical approach. I think machine learning people need to take more cues from biologists and really observe what's going on first. There's one extra interesting thing about this. You brought it up just now. You saw how these embeddings seem to be invariant to rotations or scaling. There's an interesting potential approach here, which in principle could find ways to maybe decode people's information from the vectors. So if you're a provider of, say, cloud vector storage or something, and people are sending you their data, it might be a risk of you being able to read that and people don't want to know that. But most of the operations that you perform in vector space are, like you mentioned, invariant to translation, rotation and scaling. But that means that you can put your, even if I know that your data came from, say, OpenAI's ADA-002 model, if you apply a random transform and maybe add some noise, it's very, very, very difficult for me to recover that transform, but I can still perform the same operations that you want me to perform in that space. We haven't evaluated this in-depth. It's an idea that I had fairly recently. But you can think of this as a form of homomorphic computation, because I can perform computations for you without really knowing what your data contains. And I think that's a really interesting thing as well. And that might be a whole direction on its own in a little while.

Nathan Labenz: (29:01) I mean, the security, I'm working on a couple projects where I'm like, boy, that actually could be a great solution to an otherwise thorny problem of where exactly are we sending our client data and who is seeing it in what form. So, yeah, that's a really interesting insight and a great connection. I mean, the convergence of so many things is just striking to me. We just did another interview with the authors of BLIP, the computer vision model. They just came out with BLIP-2. And BLIP-2 reduced the training computation requirements by 95%. They were able to train the whole thing on a single machine in under 10 days. And the reason that they were able to do that is because they created an ensemble approach where they had a pre-trained vision model and a pre-trained language model, and they just trained this connector model between the two. I think they said it was 200 million parameters. And what it predicts are text embeddings that are derived from the images. And so then they're able to inject this representation of the image, skipping the embedding layer, just going straight to injecting the embeddings as kind of a prefix to the prompt in a way that is, the thing that I thought was kind of most fascinating about that is they're accessing a part of that embedding space that no text could, in fact, access. We talked quite a bit about, like, is there any way to sort of reverse that to, like, what text would you have had to put in to get those embeddings? And they were like, yeah. There's really no text that would create similar embeddings. We're just finding these weird spaces, but the language model knows what to do with it. But, you know, it's hard not to see the analogy between the vision model being sort of the eyes and their model being kind of some sort of relay layer, and then you get the language model, which seems like an executive function of some sort. You're kind of doing a similar thing. You're connecting, is that how you casually would describe it to people? The language model is the executive function or the top level processing, and then you're helping get into deep memory?

Anton Troynikov: (31:20) I think that's a reasonable analogy, yes. I think another paper that's in that vein, which just struck my mind, is Robotics Transformer, RT-1. It came out from Carol Hausman's group at Google Brain Robotics. And what they do is, yeah, they have a perception subsystem that sits on top of everything, and then they have this planner, and the planner just basically learns from these multi-head embeddings to output plans and actions. It's really, really cool. It's kind of similar to the system that you described. It's all coming together pretty fast. It's really an interesting time to be working. I think that there's also just untapped veins of research here. We noticed this very early on, before we started Chroma, that the incentives between, let's say, academic ML AI research and industrial AI research, and I don't mean universities versus industrial research labs, because they're mostly doing similar work now. I mean more the sort of work that production machine learning deployments are interested in versus the sort of work that maybe pushes you towards AGI, is fundamentally different. And one of those big differences is in academic research, there are accepted community benchmarks, and your aim as a scientist is to demonstrate the performance of your model on these accepted benchmarks, right? But the benchmarks are static. Whereas in the real world, the data is always changing. And questions of like, should I train my model? What do I train it on? Is it working better or worse? Monitoring it, measuring it, are of much more salience than demonstrating performance on benchmarks. We founded Chroma with that observation in mind, that there's actually a lot we can do and mine some of these rich veins of research if we just focused on them in the first place.

Nathan Labenz: (33:04) Going back to approximate nearest neighbors, we talked about various tunings you can do where you can pre-compute some stuff depending on your workload, where you can learn these transforms to get more and more relevant over time. Do I still have to worry if I'm doing that that there's some sort of systematic blind spot in my system? Like, if I'm, you know, just to start to bridge to Stable Attribution. If you've applied all these techniques and I upload an image that I got out of Stable Diffusion, is there any way that we could start to assess whether or not there's some sort of systematic bias there? Is there some part of the latent space that's being kind of unfairly not brought to the fore in the attribution?

Anton Troynikov: (33:55) I think so. I think so. And actually, again, there's an experimental feature we're deploying into Chroma very shortly, which will tell you the relevance of what's been retrieved. So you can actually do something about it. You probably know there's this two-step retrieval process as well. It's called HyDE, where instead of just embedding your query, what you do is you send your query to a language model, you let it hallucinate whatever it wants, a response, but it looks more like a response. Then you embed that and do the query. But it's expensive. It's expensive to do two large language model calls. We did some napkin math not long ago, and we figured out that, for example, one company who we've been talking to, if they did it even the vanilla way, it would be hundreds of thousands of dollars per week. So reducing the cost of this is actually really important. And so what we realized pretty quickly was we could make this HyDE approach more efficient if we could algorithmically figure out when the results would be relevant or not. And so we're deploying that very shortly, and it's a very general idea that I think people can also integrate into their app development flow. Again, because all this stuff is geometric and nicely continuous, we give you a probability instead of just a threshold yes or no. So, yeah. But, again, pretty much everything we're talking about, more research, more work needs to be done.

Nathan Labenz: (35:11) Tell us, give us the brief overview of Stable Attribution, and then, you know, those who want to go deeper on that topic specifically, definitely recommend the show you did with Packy on the Not Boring podcast feed. But give us the kind of short overview, and then

Anton Troynikov: (35:27) I have some questions that you didn't get to cover on that conversation. Absolutely. So the short version is fairly straightforward. We saw a need and a possible technical approach to figuring out or at least starting to figure out how images in Stable Diffusion's training set influence particular generations that are created. And the way that we did that is by doing various, by extracting various information, both about just similar images to a given generation in the training set, but then also figuring out what the right similarity is. The reason we took this approach is, again, because there are optimal approaches, but they are all computationally intractable. For example, one more optimal or more principled way to do this might be to remove examples one by one, and retrain the model every single time, until the image has gone far enough away, then you say, okay, those images influenced the most. But of course, every training run of Stable Diffusion costs about $600,000. So it's computationally infeasible. We found a different approach. We think it's principled, and we applied that. We leveraged other, you know, various properties of how diffusion actually works alongside similarity in latent space. We construct what we think is a principled similarity metric and go from there.

Nathan Labenz: (36:41) Cool. So give us just a little bit more of the principles behind it at least, and then I'll get into a couple of questions.

Anton Troynikov: (36:48) Yeah, sure. Assuming the audience here knows basically how latent diffusion models work: they're trained on image-text pairs, and they're trained to reproduce the same encoded latent as the training example through the diffusion process. They're trained by iteratively adding noise to the training examples' latent vector representation, and then training a network that reverses that noise. And what's great about it, what's actually the big breakthrough, is this is very efficient, both in terms of how much data is used by each dimension in the vector, but also because you can train in parallel each denoising step. That's the big thing. So there's a few parts to this. The starting point is: given that the model's training objective is minimized if it can exactly reproduce the latents of all of its training examples, and because we know that the latent space is continuous and smooth by definition—because that's how backprop works, you have to do that—then it's not unreasonable to say that training examples that are near the generation in latent space in some way are influential to the output of that generation. So that's where we started from. We took a look, we found out a bunch of interesting stuff. We found that, for example, there were a bunch of papers showing that stable diffusion actually reproduced exactly or very close to examples in its training set, which people denied for a long time. Don't know why. Now they tell me, "Oh, you can reproduce a tiny number of examples." It's not what they were saying before. But anyway, we found that noise in the training set is definitely something quite interesting here, because the CLIP representation of images and text might differ for a given example. And you can imagine why—the same image might have a lot of different captions, even in different languages. And then a lot of the data is just noisy. So we took that into account. And we took into account the attention mechanism of the diffusion model itself by constructing attention maps. And attention maps are actually, if you look at them in a slightly different lens, they're actually just a matrix multiplication. They're a linear transform of that vector. So when we apply the attention map, it's as if we're transforming the latent space, and then we can do similarity search in that new latent space. That's a very short overview.

Nathan Labenz: (38:55) Cool. So yes, again, for more depth on that, go to the Not Boring "Anton Teaches Packy About AI" episode six, and there is quite a bit more detail there. So just kind of zooming out from the technology a little bit, I naturally went online and tried it, and I tried it with a mix of images that I had made with stable diffusion. I know that it's not really meant to be used on natural images that I took on my phone. But of course I tried that as well, and it does work. And either way, you do get images back that are similar. The divergence definitely between the real images—stuff on my phone—and what comes back is bigger. And I think you have an explanation for that around why it doesn't come back with such similar images even though I mentioned that they must exist, right?

Anton Troynikov: (39:52) Yeah. So here's the interesting point. The similarity is not perceptual human similarity. It's how the model perceives the difference. That's an important thing to note. When I say influence, it's not necessarily the same way that an image might influence a human artist, where they would draw motifs and inspiration from it. It's more this very mechanical, raw interpretation—these vectors are the same, right? These vectors are similar. So to a human perceptual system, once those images are decoded from their vectors, they might look pretty different, and you won't know what's similar about them. But we know for a fact that ML models are kind of weird in the way that they think, quote unquote, about the world. And they might represent something quite differently to what a human would. They might focus on quite different things because they are mechanistically pursuing an objective. They are not trying to encode any meaning. And I think that's what produces surprises. I think one way to think about what you get back with inserting images that are not from stable diffusion is: if stable diffusion had produced this, if it were possible for the model to produce this—or put another way, if the model had produced this image with a corresponding caption that was appropriate to it—here's what would have been in the training set. It's a long chain of counterfactuals. It's not really meaningful. So here's the thing, right? And you can go and look at this, anyone can. If you go to GitHub and you look at the stable diffusion code, whether version one or version two, as provided in the open source repo, there is a watermark function in there. That watermark is not present in most hosted versions of stable diffusion, and we don't know why. Or at least if it is present, people don't tell us what the watermark is, and we don't know why. That was our plan originally to reject non-stable diffusion generated images, but it's just not present. We felt that it was better to release and get feedback than not.

Nathan Labenz: (41:45) Yeah. Actually, I think there's an interesting—it is an interesting experience when I used my real, taken-on-my-phone images. The results were kind of—I don't want to use this term, but it's the only one that's coming to my mind. I know it's loaded. But it almost did feel like the pieces that were kind of collaged together to make my actual real image. I tried one where it was just my kids saying goodbye to their friends at our front door, and what I got back was not all kids at front doors, but you could see these elements that were very resonant across the images. And I felt like there's almost something here product-wise that is kind of interesting. It's a sort of diverse image-to-image search that picks up on these notable aspects.

Anton Troynikov: (42:46) Have you seen, I think it's Same Vibe or something like that? It's essentially that. It's a mood board search engine that I think uses similar principles under the hood. I think there's a bunch of ways that you could take this. I'm actually curious—if we do keep doing experiments in this direction, maybe we will. Right now, obviously, we're focused on other things. I think this is a great chance to turn generative models into something besides a one-way process. If you can manage to do these kinds of relations, you can really start to use them more in a search or an iterative conversational search type of approach, which I think is really interesting. But yeah, the thing that you're getting back is: okay, this is how the model sees your image, almost. This is what it would have tried to do if it was generating it.

Nathan Labenz: (43:37) It is cool because it does have this sort of—it sort of reflects back to you, like, these are the things that made your image distinctive in a way that you couldn't necessarily have articulated yourself. But oh yeah, I see my kid in this picture is turning over his shoulder and looking a certain way. And this other image is quite different in all the other respects, but it has that same pose. So there's some feature that they have very much in common, even though the other features are very much not in common. I actually would recommend that people go try their real images on your search. That's not what you necessarily meant it to be.

Anton Troynikov: (44:19) Please don't yell at me if you do that.

Nathan Labenz: (44:22) Yeah. FYI, it's same.energy.

Anton Troynikov: (44:26) Same.energy. I really like that.

Nathan Labenz: (44:28) I think it's fun. Okay. So let's return to the intended use case. So you put an image that you made with stable diffusion into the product. It does its searching and principled extrapolation of the search, comes back with images. Sometimes I see a few. Sometimes I see a lot. I was really left wondering, how are you deciding how many to include? Is there a certain cutoff, or are you kind of doing a top-p type of thing? What do those distributions look like?

Anton Troynikov: (45:02) Yeah. That's a really interesting question. It's something we've tried to instrument, but it's, again, one of those things that you need a lot of data to figure out, and a lot of compute, which we are not applying to this right now. But it's interesting. You're right. Sometimes you get not many matches, and sometimes you get quite a lot. And I think that depends on where you land in this modified similarity space that we talked about. Like you mentioned earlier, with the language model where it can generate reasonable text from places that are nowhere in its training set, you can think of this almost as a dial maybe from the model reproducing stuff to the model generalizing, right? If you can maybe look at that from that lens. And the thing is, the stable attribution algorithm really doesn't take model generalization into account. We're leaning really hard on this reproducibility rather than generalizability, and there's a lot of questions to explore there. And I think that they will get explored because we're going to look for ways to continuously improve the performance of the models outside of the scaling laws. Although a lot of the labs who have the most compute will keep pushing on the scaling laws. The labs who are looking to just do something interesting, or even at the product level, people who are looking to do something interesting will experiment with stuff like this and be like, "Oh, okay, I noticed that this part of the generative space doesn't have much in it, but people are asking for a lot from here. Maybe we should go out and get stuff that people want from this part of space and then get people to actually make images for it." It's an interesting possibility for the future.

Nathan Labenz: (46:38) So when I do get those results, is it fair to say—I think you said this on the Packy show—that everything influences everything in this space, right? Because everything's involved in gradient descent at some point. So in theory, if you had the fully principled answer, they would all be infinitesimal but probably non-zero.

Anton Troynikov: (47:07) Yeah. It's an interesting question. I'm not sure that's quite right. I don't think that they would necessarily be infinitesimal. I think that different generations will draw from different parts to greater or lesser degrees because of the training objective. Further, things that are further away should naturally not influence the generation from a particular conditioning caption as much as things that are related to that conditioning caption. I think that's also a pretty interesting question because one open question is: in what way does generalization actually influence the output? We haven't looked at that. I think that you could do this kind of research with a toy model and really see, oh, you know, just have a tiny diffusion model make pictures of flowers or something. And actually just literally just burn a bunch of compute on figuring out: this is proportionally how the dataset actually does influence things. I think there's another really interesting approach here which you can take with a toy model, which is basically compare the stable attribution approach with the gold standard and see how good or bad we actually are. If we had GPU time, we would just do that. And then I'd be like, "Okay, yeah, everybody was right. This sucks." Or I could be like, "Oh, actually, you know what? This is really great, I have numbers now." But until then, these questions are pretty hard to answer.

Nathan Labenz: (48:26) So in the processing running today, does it generate a score for each image that it finds? They're ranked, right? Does that have a clear relationship to the whole? Do those sum to one? Or do they sum to anything that you know what they sum to?

Anton Troynikov: (48:46) I mean, there's some metric in space. We treat it more like an ordinal thing. And then you sort of—you can get a proportion out of that by, say, you take the top end, and then you divide evenly according to the similarity metric, but it's not that meaningful. It's much more of an ordinal question. You can say, these are closer, these are further apart.

Nathan Labenz: (49:08) So then what is happening when there are no results? You also have a threshold below which you just are like, nothing is close enough to this?

Anton Troynikov: (49:15) There's a few filters going on behind the scenes. We have a threshold where we just stop looking. You know, it's pointless to get things that are far away. We say that, okay, really only this condition should be regarded as influential. It's an arbitrary choice. It's a heuristic. And again, that's like, okay, well, here the model is really extrapolating and filling the gaps.

Nathan Labenz: (49:34) We'll leave aside for the moment a deep dive into why you did this - the vision for an economic dimension to this attribution that could one day come to exist. You talked about that in the recent show, so I don't want to rehash it. But I wonder to what degree this is separable. I was just running this thought experiment where I'm like, okay, let's say I have a Stable Diffusion image and Greg Rutkowski is in my prompt. It's amazing - you go on these sites and just look at the prompts, and it still feels like one out of every three is Greg Rutkowski by name.

Anton Troynikov: (50:14) I think it got stuck in people's heads early on. It's amazing how the very early things that people discover just become gold standards, even though it could have been anyone. It could have been any choice. There's plenty of concept artists working today. It's just one of these odd artifacts of Internet culture.

Nathan Labenz: (50:26) I guess I'm wondering, if you were able to kind of peel back - and they're doing this now with Stable Diffusion, they've got this opt-out mechanism and all that kind of stuff - to what degree is everything sort of meshed together and not really separable? A thought experiment would be: let's say you took all the stock photos and then you added all Greg Rutkowski's real art. Do you have any intuition for whether I could still get Greg Rutkowski stuff out of that, or do I need a richer, denser space?

Anton Troynikov: (51:03) Yeah, look, this is exactly the kind of research that needs to get done. This is exactly the kind of research that hasn't been done to this point because we've been working towards capabilities of these models, and we need to start looking at controllability and engineering principles. The way that I think about this is: in the early days of aviation, we had intuition about what makes airplanes fly. The Wright brothers were wrong about how the Wright Flyer worked. They just had intuitions about it. It flew - that's an objective demonstration. It doesn't really matter if your intuition is wrong if the thing works. And then there's this Cambrian explosion of different types of aircraft. People were putting two or three wings on them. People were experimenting with different materials, different fuselage shapes. You've got things that look like birds, things that look like crazy fans in a tube, because all we had to go on was intuition. We didn't have engineering principles yet, and we didn't even really have the tooling to develop those engineering principles. That's where we're at today, in my opinion, with machine learning. We have a whole bunch of intuitions, we kind of know how things work, and what's really interesting - and this happens very frequently - is some paper will assert, even with Stable Diffusion, even with latent diffusion models, they were asserting it's like Langevin thermal diffusion dynamics. And a paper came out the other day, I think it's called cold diffusion or something like that, where they're like, "No, it's not. Here's an objective demonstration that that's not true." We're in that era of just experimentation. None of this stuff is really well understood, and we're seeing the early days of people trying to develop principles. The scaling laws were the first thing, where it's like, okay, we're going to empirically demonstrate that you need to increase data as much as you increase compute, otherwise it's worthless. That'll develop further until we probably get to a point where we can actually design the things that we're building for the mission that they are intended to perform, like we design aircraft today. And we still don't know everything about flight. Flight is a very complex discipline. But we now have the tools, and we have enough experimental evidence, and we have enough laws of physics that we can really build pretty amazing aircraft. But it took a long time.

Nathan Labenz: (53:09) Yeah. For what it's worth, it sounds like you might have read it, but the David McCullough Wright Brothers biography is pretty short and absolutely fantastic. I could not recommend it more highly. It's very striking just how these two brothers cared about basically nothing but doing the work and their family. But they really did not - there were a lot of folks out there who were hyping themselves up as the great experts in flight, and they never actually flew anything despite their supposed expertise. And these two dudes just put their heads down and did the work until they actually had the goods.

Anton Troynikov: (53:55) Yeah. And then they milked it through intellectual property law just as far as they could until Curtiss bought them. It's kind of a Greek tragedy, honestly.

Nathan Labenz: (54:04) It's interesting. That's not as much covered in the book, so maybe I'm missing some of the less flattering aspects of their story. I don't know if you have any take on the technology angle. I was thinking also as I was doing this, you go through all these prompts and you just see 4K, Unreal Engine, Pixar style, RED brand cameras. And envisioning your kind of imagined future of people getting some sort of economic remuneration for their contribution to this, do you think that those sorts of technology layers also have a claim?

Anton Troynikov: (54:49) It's interesting. That's a really interesting point. So first of all, for me, as somebody who works in this discipline, prompt engineering kind of feels like - and I've said this - it feels like trying to pick a lock with a wet noodle. There's a behavior that you want the model to do, but the only interface you're using to the model is literally typing text into it, when what you really want to be doing is having feedback mechanisms inside it, like with control systems like we have in industrial machinery or an aircraft. So what actually is adding "4K" doing? Open question. Nobody's answered it. This is more of that kind of research that I'm talking about that needs to be done. Why does 4K matter? It's kind of like with InstructGPT - why does telling it that it's the world's expert in something make it produce better answers? What's actually happening? The danger is anthropomorphizing these models. It's like, "Oh, it's thinking better." It's not a person. It's a machine. Dissect the machine, see why it's doing that. Where is it going in its latent space? All these models are encoding and decoding the latent space. To answer your next question, how do you attribute rights holders to equipment? I mean, that's fairly settled. You don't. Although, you know, it could be interesting. Personally, I'm not a huge fan of adding additional copyright law or strengthening derivative work IP law. One of the reasons that we did this project was we didn't want Disney to own everything forever in the inevitable copyright backlash. As you can imagine, okay, derivative work copyright law gets strengthened, and then the next day, Disney says, "Well, anything vaguely superhero-looking is in the Marvel Universe, so I'm sorry, you can't draw pictures of that anymore and put them online." So that's one of the reasons we actually released this. It's actually aligned incentives here. I hadn't thought about that angle. I think traditionally the medium is elided. I don't think that people really have claim to images based on the equipment used to capture them. Could be different here. I don't know. I can't say that I have an opinion.

Nathan Labenz: (56:52) Yeah. I don't really either, and it definitely seems quite settled when it comes to photographers owning their work - they bought the camera, and that's kind of the end of that transaction. But this does feel like it could be different. Certainly, just the number of times you see that as you go through the prompts does kind of suggest, gee, there is something to that that is highly desirable that they've kind of created that people have a hard time getting without them. So I wouldn't be surprised if RED camera makers attached to something at some point.

Anton Troynikov: (57:29) Look, I mean, I think maybe it's something similar to how photographers will share their equipment. They will say, "This is how I captured this photo," right? They still captured the photo, but there's some credit to the equipment they use so that other photographers can learn from them. I don't know. Could be. Yeah, it's a lot of free advertising for them right now as well, so they might not want to opt out.

Nathan Labenz: (57:47) Prompts are like this first, the zero stage of attribution. If you're typing Greg Rutkowski, you should probably credit the guy somehow.

Anton Troynikov: (57:52) Right.

Nathan Labenz: (58:01) The role, the importance, the problems associated with noise - I think this has been one of the concepts that I've been thinking about the most since I listened to you talk to Packy. You said that, for one thing, these massive web-scale datasets have a lot of image-text pairs where there is high similarity, and that basically means the text describes what is in the image. And then you have a lot where it's kind of a totally different thing that may have almost nothing to do with what is actually in the image, and that just serves to add a ton of noise to the dataset. And then you're kind of extending that to say that noisy portion of the dataset is helping it generalize. I don't have a great intuition for that, so I'd love to understand your thinking there a little better.

Anton Troynikov: (58:54) Yeah. So in, let's say, old-school machine learning - even though it's been around since 2015, in the prehistoric times, in the times of our forefathers - one thing that people noticed very early on was that neural networks were very prone to overfitting. And the way to prevent overfitting was to inject noise into the model's training loop. So what you would do is dropout was a very common thing, where you just disconnect connections between neurons while it was training, so that it didn't end up relying on any specific connection too much. Or, you know, all these little tips and tricks and techniques, which kind of got obliterated by just scaling - just make the model bigger, feed it more data. But at the time, this was a huge issue, right? And so the way to maintain generality, so that the model, once you evaluate it on your test set, still performed well, was to inject noise. You'd fuzz the images, you would flip them around, try to do plausible things that might exist in the real world. And so that's the analogy that I draw here. Noise seems to lead to generalization in some way. It seems to allow it - when the text and the image are not very related - to grab something conceptually from those images. It's like it almost picks supporting structures out of them without necessarily attaching the associated meaning. That's just an intuition that I have. I can't say for sure that it's real. Another way to look at it is: you can imagine one image, many captions. It allows the model to come to the same image for many conditions. Imagine you only actually had one image, but you just train the model on one image with many, many different captions. It would always end up in that image, regardless of what you put into it. Maybe that's also helping weight the kinds of generations that you get. It suddenly is able to connect one image concept with many textual concepts. Things like that might be helping it. Again, this is exactly the kind of research that needs to be getting done. Well, hopefully. I want it to get done. I don't know if other people do.

Nathan Labenz: (1:01:04) It's a target-rich environment right now, that's for sure. An hour in and I am able to - that's almost like my KPI for this show - how many times I can get our guests to say that research needs to be done. I think we're at three, so I'm doing alright today. So just briefly going back to your product with Chroma and the vector database. You said at the top, there are all these different kinds of things that can be represented as vectors. So we've largely talked about text, which is going to drive this retrieval future where the AI is going to have seemingly photographic recall of all of our emails and our interactions and so on, and even better than that, put that information together in a way

Anton Troynikov: (1:01:48) that actually answers the query that you have, answers the question that you have, or fits the context that you want it to operate in.

Nathan Labenz: (1:01:54) So what else is getting embedded these days?

Anton Troynikov: (1:01:59) Oh, boy. I mean, you guys have probably heard of Whisper before. You have a podcast. So yes, Whisper is audio embeddings. Basically any modality that you can stick a neural network onto, and preferably a transformer at this point, you can get embeddings from it. So we've got audio already. We've got video. I've been speaking to a couple of people who have been working on that. It's actually starting to get commercialized too, like video embedding. So you can do semantic search inside a video. You can be like, find me all the footage from this party that's actually fun to watch, and it'll just do that for you. So video, and I think it'll just keep going. I think that we'll see over in robotics, for example, chains of actions are also embeddable. Like, the person performed this task. That's embeddable. Lots more like that. Basically, all kinds of modalities will continue to get embedded, and I think what will continue to happen as well is, so we've talked about CLIP, contrastive language image pre-training. We'll see more models trained that way. So right now we have Whisper, which is text to audio in the same space, and we've got CLIP, which is text to image in the same space, and we'll have video and text in the same space. But then you can also think of audio to video. You can think of images to audio, that whole matrix of things. And each one of those entries in that matrix is a potential application of one kind or another, which is a

Nathan Labenz: (1:03:18) really exciting thing. Are you seeing things also from biology starting to get embedded?

Anton Troynikov: (1:03:23) I was thinking genomes and protein structures. Yeah. Biotech, I think, is waking up to this. Biotech is a very interesting field to think about this in, because in biotech, you kind of have to get it right. It's okay if your semantic search returns things that only kind of look like the thing that you wanted. In bio, you really need the protein to be the one you were looking for. But the other part of this is in bio, most of those embedding functions that we talked about at the top of the recording are still hand-rolled. Very few are actually trained and modeled. I think we've seen the success of this in these other domains that we've talked about, and I think biology is starting to wake up to this whole possibility. And so I think in the next year or two, we're going to start seeing a lot more work in this direction. And I've been talking to biotech founders and companies and people working in this space, and they're all very interested to see what can be done here.

Nathan Labenz: (1:04:20) You know, I've been using the phrase Kurzweil's revenge increasingly often over the last year or so, and that's probably a good bridge to the last topic for today.

Anton Troynikov: (1:04:32) Are we all gonna die, or what's gonna happen? Yeah, that's a good bridge. That's a really good bridge because you brought up Kurzweil. And one of my favorite things about Kurzweil is I went on the website not long ago, and it was the tenth anniversary of The Singularity is Near. And that just made me laugh. So look, I don't think we're gonna die. I have yet to see a convincing argument that we're all gonna die. Every argument that I've seen in that direction requires some sort of system of basically incalculable and definitionally incomprehensible power, which, okay, if you're gonna invoke magic, you might as well just do that at the start. I think that there are dangers. I definitely think that there are dangers. I don't think the dangers look like AI super-smart overlords pursuing human-unfathomable goals. If you're concerned with that, you should probably work on sun alignment because the sun is giant and has incomprehensible goals and is very dangerous to humanity, but it actually exists today. What I'm more worried about is, and these are cycles in history and you see them repeatedly. We see them with the invention of the printing press and then radio and then television, significantly destabilizing to society, and often very violent periods. So the Thirty Years' War is the one I always come back to, and the Thirty Years' War, in part, was triggered by the printing press. Martin Luther was able to spread his ideas very quickly throughout Europe because they could be printed, and that caused players in power to be able to attach to this movement and they produced a schism, and then parts of Europe were 50% depopulated in a matter of a few years, which is incredible. It's still, per capita, the Thirty Years' War is still the worst conflict in European history, per capita. And then you have the rise of totalitarianism, which was, you put the radio, you put a radio in everybody's home. Suddenly a person who wants to be dictator doesn't have to walk physically from city to city or just convince you through text. You could hear them on radio. And I think it takes time for societies to develop basically what I think of as mimetic antibodies. I think if someone heard Hitler ranting on the radio today, they'd be like, damn, this guy's crazy. Although, you know, we do have that. There's a few people who are still pretty persuasive. But it takes time. It takes time. And there's also this bizarre thing that occurred in my lifetime, which still deeply confuses me. Back when the web first came out, everybody kind of knew not to trust it, because people were like, oh, anyone can put anything on there. Don't trust anything you read on the web. People can just lie. And that has never changed. You can still just go on the internet and lie. There's that famous meme macro. But people seem to have regressed. People seem to believe what they read online more readily now, and it's almost because it got legitimized as an information distribution mechanism that people lost that immunity. And the reason that I bring this up in the context of AI is because you can imagine very concrete use cases for even something like ChatGPT, where you can have finely targeted, down to the individual, propaganda. We've seen the success of social media in doing this, ad targeting being able to be really mobilizing for elections in the United States and in other countries. I think at the point where you can create text that's individually targeted and kind of start hacking people's actual preferences to get them to vote for you or to incite violence or any number of things, that's kind of dangerous. But it's not the AI, it's the people running the AI. There's another interesting thing, which I've been speaking to a few people about, and I'll mention in the abstract, which is there are dangerous things in the world which are still gatekept because the knowledge of how to do those dangerous things is very difficult to come by. But if you have this reasoning machine in your pocket, you don't have to be smart anymore. You can ask the smart reasoning machine that you have to do the dangerous thing. And so that's another real possibility. I called it a knowledge capability gap somewhere. And I've been trying to get ChatGPT to tell me how to make a neutron initiator, which is a classified component of a hydrogen bomb. It won't do it. These things never let me get my work done. These are things that are more worrying. And then these are just the things we can see. I think the destabilizing effects of new media technologies are always unexpected as well. And I think we're in a very early stage, so we need to be vigilant about this stuff. I think that's the perspective. I don't think we're gonna die, but I think it might be dangerous and in the cursed sense of interesting times.

Nathan Labenz: (1:09:15) Yeah. That seems to me, that's kind of why we called this show the Cognitive Revolution. That seems to me kind of baked in at this point. From my survey of the AI landscape, we've talked from a couple different angles about how the same architectures are working for everything. All this progress is happening in parallel. And we're talking to folks like yourself who are chipping away at it. And I don't think, by the way, I don't think anybody really thinks right now that the core technologies, the fundamentals, are gonna stop right where we are, or that there's gonna be no new insights or no architectural improvements. I expect that too. But even if I kind of rule that out, I think, geez, the core stuff right now seems powerful enough to be transformative. It hasn't been productized. It hasn't been sanded. The rough edges are still there. They haven't been sanded down. It hasn't been connected to all the other things, really. New Bing, you know, I'm still on the wait list, for God's sake.

Anton Troynikov: (1:10:30) You missed the fun part. I know. Well, yeah, for better or worse. But these things are all gonna happen, right? And so the impact, to me, seems pretty baked in. I think we're in for a wild ride that we're really not ready for. I don't think there's ever been a time in history where someone's been like, okay, guys, get ready. We're gonna press this button, and everything's gonna change. But I think that actually speaks to our resilience as a species. That's still literally true. Every single time that's happened, we've come through it. And some people take the perspective that this is a unique risk. I think that the actual risks that exist rhyme pretty strongly with other times we've faced them. So I put that not as a general statement like, oh, we'll be fine. We've always been fine before. It's like, no, I think we'll be fine, because the actual risks we're facing rhyme with risks we have actually faced before. What you put right now, I think, is actually a good point, and people working in, broadly speaking, the realm of AI and AI safety. Sort of the consensus is even if we stopped training and developing new models today, we'd keep finding new uses for the models we have today. If you see how people experimented with GPT-3 and 3.5, discovering that you could ask it to think step by step makes the answers better didn't require training a new model. You're just literally like, oh, this thing can do that. They call it a capabilities overhang, if you're in this sort of safety space. So yeah, we could literally stop training right now and still not know everything we can do already. Yeah. There's a lot of capabilities overhang.

Nathan Labenz: (1:11:45) The capabilities overhang is vast. I'm really uncertain whether or not we're all gonna die, but I definitely don't rule it out. And I'd like to hear your kind of response to, you know, I'll just give kind of a superficial, high-level version of it. But I would say, if there's a canonical argument at this point, I would say it's probably Ajeya Cotra's post, which is called, without specific countermeasures, the most likely path to AGI likely ends in disaster. Could you

Anton Troynikov: (1:12:24) also link me to that post after this chat? I genuinely read a lot of this literature. I read Eliezer Yudkowsky's thing, AGI Doom. I've read, I think people are always surprised by how much LessWrong I've read and how much Alignment Forum I've read, given what my position actually is. I'd be curious to read that, but what is the argument that is advanced there?

Nathan Labenz: (1:12:48) Well, there's a lot. So you know a lot, and they can be subtle. But the way that I like to summarize it for folks is, you know, what she describes as the most likely path to AGI at this point, she calls human feedback on diverse tasks. So kind of

Anton Troynikov: (1:13:03) a

Nathan Labenz: (1:13:03) generalization of the reinforcement learning from human feedback to more and more different kinds of tasks and likely multimodal types of tasks. The challenge that she sees that does seem to me really hard to dismiss, not to say that I'm sure that it plays out this way, but I really can't see a great reason why I should be confident that it won't, is just simply that we are not reliable raters. The people doing the evaluation have these flaws, and we know from the heuristics and biases literature, the cognitive science, we know that people are flawed. In fact, there's a

Anton Troynikov: (1:13:44) Richard Ngo from OpenAI has this great paper called Alignment from a Deep Learning Perspective, and he brings up a lot of these great examples, actually. He brings up examples where the model actually fools the human rater because it looks like it's performing a task on camera, but it's actually just faking it. So yeah, okay, so far, so good.

Nathan Labenz: (1:14:02) At some point, it seems like these models are getting pretty smart already, right? They have increasingly, it's now in debate whether or not they have effective theory of mind, so on and so forth. It seems like it's not too much of a stretch to get to a point where the models will maximize their feedback score by not straightforwardly, truthfully, honestly doing the task, but instead start to develop a model of, well, here's what reality is, but here's what this person is likely to give me a score for. And so then you kind of create this incentive for the thing to begin to deceive. And if you create that in an environment where we don't have the interpretability chops right now, obviously, to detect if it's there, certainly with any reliability, then it seems to me like you really are playing with real fire.

Anton Troynikov: (1:14:58) There's a leap of logic we made here. So we have a model. It's performing a task that's valuable to humans. We're rating its performance on the task. First of all, if it's performing the task that we asked it for, and we're rating it on that task, what is the difference between it doing it deceptively or correctly if we can measure the outcome? And I don't mean in the sense that, oh, human raters, et cetera. I just mean in the sense that it's either doing the thing we want it to do or not. And so in the deceptive examples from that paper that I mentioned, yeah, okay, it wasn't really raising the ball, but also there was nothing riding on it raising the ball or not, except our benchmarking of this model. That's the first part of it. In engineering, the real test is, does the machine perform the task? Instead of just in a test environment, but is it actually doing what it's supposed to be doing? And this is where I come from. The thing that people seem to call alignment in AI research is called control theory everywhere else. And these things are dynamical systems that can be understood. They have actually some really nice characteristics that allow us to understand them as dynamical systems, and we know how to build controllable dynamical systems. That's why the F-16 can fly at all, is because we know how to do this. The other leap of logic that's being made here is, okay, well, it's deceptive to human evaluators. It can hack their reward function. How does it follow from there that we're all going to die? What is the path to that actually happening? I've never heard a clear example of this that doesn't invoke some sort of other form of calamity that if you're worried about, humans could just do that. You don't need to worry about the AI. You can just worry about the calamity itself. I get this thing where it's like, oh, it develops an undetectable nerve toxin that triggers once it's implanted in every human being on Earth. Well, okay, so why not a human with a machine that's optimizing in the same way produce the same outcome? Or, you know, this thing that it builds like these nano self-assemblers, and there's this famous post, create an identical strawberry thing, where it does these nano-assembler things. Okay, but first of all, as far as I can tell, nano-assemblers are science fiction. A lot of people like Drexler. I haven't actually seen a demonstration of this technology. But second of all, you don't need the AI to be worried about the nano-assemblers at that point. So it's not clear to me that an AI that reward hacked some humans is any more dangerous than an out-of-control bulldozer. We know how to deal with this stuff. We ought to think about it in those terms. I think the real danger here is anthropomorphizing the system and saying, no, it's thinking it has goals. It doesn't have any goals. It's a computer system. It can be anthropomorphized to have goals, but concretely, it's doing matrix multiplications and applying nonlinearities to them. And that's a nonlinear dynamical system that's continuous and differentiable, and we know how to deal with those. It just doesn't strike me as a credible argument. It only strikes me as a worrying argument when you start to turn these machines into people.

Nathan Labenz: (1:17:59) I worry about the whole spectrum, and I get ambivalent on this in the classical sense of having strong feelings both ways. I love working with all the AI technology. I love studying it, and I'm super excited about the upside. But, you know, when you talk about a bad actor with an AI, that's very much, I think, of concern, no doubt. There's a project out there right now called, I can't remember exactly what it's called, but it's basically text to regulatory RNA sequence, which starts to sound pretty insane. I asked a model one time to write some hard science fiction based on that premise, and it got me to some pretty strange places pretty quick. Obviously, we have so much time. We can only go so deep on this today.

Anton Troynikov: (1:18:49) There's one example of that. There was that famous paper where they asked it to produce effective pharmaceuticals, and then they turned their reward function around and just started producing deadly poisons almost immediately. That stuff is more concerning. Ultimately, in order to kill us all, whatever does that has to obey physical law. It's a large leap for me in the physical world, especially, and I think this comes from my background as a roboticist, in that it's really hard to do things. It's a lot easier to do things with information than it is to do anything in the physical world, regardless of how often people analogize things like, oh, DNA is just like binary encoding, or CRISPR is like programming biology. It's not really like that in the real world at all. To give another analogy, anyone can make sarin gas. The problem with sarin gas is not making it, it's deploying it. It's getting it into place. That's why we don't have constant sarin gas attacks. Whatever this thing is that's supposedly going to kill all of us has to be able to act in the physical world in a way that I just haven't seen any system capable of doing yet.

Nathan Labenz: (1:19:55) Well, I hope I never meet a robot that has the regard for me that new Bing has had for some of its early users, that I can say for sure. So we're running out of time. I would love to check back in, you know, at some point down the line and see how both your company is developing, whether or not you've continued this stable attribution project, and we can also check in again on how much existential confidence or nervousness we have. But a couple just real quick hitters to close us out today. One, AI tools in your life. What are you using? What has changed your workflow? What do you recommend?

Anton Troynikov: (1:20:36) Yeah. ChatGPT has sped up my programming work a whole bunch because whenever I work with an unfamiliar library or get a weird bug, I ask ChatGPT. Sometimes it hallucinates stuff, but at least it helps me look in the right direction. I use coding assistance. I've been using Copilot. I recently tried Cursor. Both are pretty good. Definitely sped me up as an engineer, mostly because I no longer have to Google or go to Stack Overflow or look for documentation of stuff that I'm unfamiliar with. And what that means is it's not replacing me as a programmer. It's getting the stuff, it's getting the things that I really don't know how to do out of my way so that I can focus on the things that I'm actually really good at. That's been a huge, huge productivity boost. Yeah. I think that's really the main ones that I'm using today.

Nathan Labenz: (1:21:19) Also recommend Replit and their Ghostwriter and Ghostwriter Chat. Pretty cool.

Anton Troynikov: (1:21:25) We really like Replit at Chroma.

Nathan Labenz: (1:21:28) Okay. Quick hitter number two. Hypothetical scenario. If a million people already had a Neuralink implant and it would allow you to type as quickly as you can think, in other words, you now have thought to text, would you get one?

Anton Troynikov: (1:21:50) I think, and I've thought about this, I think if we end up living in a society where somebody makes me implant a computer in my head or it's socially unacceptable to not implant a computer in my head, I'm going into the woods.

Nathan Labenz: (1:22:01) That's a no. You're a no on Neuralink.

Anton Troynikov: (1:22:04) It's a strong no. Listen, I've seen how software is written. I don't want it anywhere near my brain.

Nathan Labenz: (1:22:09) Yeah. It's wild. I do think I kind of expect to see that day. I used to think my grandparents, who were born around 1930, and, you know, my grandmother's still alive. She's going to be 90 this year. I used to think that they saw way more change in their lifetimes than we ever would, and that perspective has started to change recently. And now I'm like, yeah. Maybe I better be getting emotionally prepared to have a hole in my head and a little implant.

Anton Troynikov: (1:22:36) Look, I know what's going to happen here. What's going to happen here is I'm going to have some neurodegenerative disease when I'm 85 years old, and they'll be like, install Neuralink, and it just fixes you. And I'll be like, goddamn it. Remember when I went on that podcast?

Nathan Labenz: (1:22:48) You'll remember it after they do the implant. That's when you'll really remember it.

Anton Troynikov: (1:22:51) Exactly. And then you'll have the memory. That'll be the real tragedy, I think. You're right.

Nathan Labenz: (1:22:55) Alright. Last one. You know, you're not so concerned about the doom scenarios. Well, give us your big picture hopes, positive hopes for, and also, to the degree that you have them, downside fears for AI for the rest of this decade.

Anton Troynikov: (1:23:11) Yeah. I think that's a really good way of framing some things. So one of the problems that we faced historically as a species is the thing that we talked about earlier, where nobody tells us, like, we're going to push a button and everything's going to change. But I think AI gives us the possibility of having tools that allow us to rapidly adapt to our situation. And kind of the fundamental thing that humans have going for us is our adaptability. So a prosthesis, an extra tool that allows us to adapt even faster, makes it easier for us to deal with problems as they emerge, and actually could make us safer in the long run. And the way that I think about that is, right now, for example, suppose that there's some frightening thing like COVID. In COVID, you needed a PhD or an MD to understand what to do with it. And there's only so many people in the world that really understand what recombinant DNA really means. I certainly don't. I'm not a biology person. But if we had tools to get people to the research front, like a PhD in a box where it was exactly tailored and individualized to you and turn you into a researcher just because you wanted to work on that stuff, that sounds like we could be much more agile and nimble as a species to deal with anything that might come up. That's a hope. Of course, the downside is every new media technology has this other property, where at the start, everyone believes it's going to be for education, but it ends up getting used for pornography and propaganda. And I think this time is no different. And the propaganda part worries me, and the sort of this new idea of knowledge capability through having a reasoning machine in your pocket kind of worries me a bit. The thing that doesn't worry me is, I don't think that you could probably reason your way to some very dangerous things, but again, those very dangerous things require you to be able to act in the physical world. So we're talking about fairly organized actors being dangerous rather than individuals. Hard to say. But, yeah, overall, I think also, one thing that's going to happen, I think one other dangerous thing which we didn't really discuss, is there's a very real possibility that another AI winter will happen. And I actually asked Sam Altman that question a little while ago. There was some sort of event. Honestly, over exuberance about the capabilities and then they failed to deliver it. That's what's happened every other time in AI winters, right? The previous one was expert systems and the one before that was neural networks. Now we're back in neural networks again. That's kind of worrying, because another fallow decade when we could have these incredible technologies is actually, I don't know, that seems really bad to me. We should try to be fairly clear-eyed. That said, I'm still bullish because, like we said, there's this massive capabilities overhang we haven't even started to explore.

Nathan Labenz: (1:25:45) That's all the time we have for today. Anton Troynikov, the company is Chroma, the embeddings database. Thank you so much for joining us on the Cognitive Revolution.

Anton Troynikov: (1:25:52) Thanks for having me, guys.

Sponsor: (1:25:56) The Cognitive Revolution podcast is supported by Omneky. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms with

Nathan Labenz: (1:26:07) the click of a button.

Sponsor: (1:26:09) Omneky combines generative AI and real-time advertising data to generate personalized experiences at scale.