Cryptography Will Revolutionize AI Data Privacy with Daniel Kang

Full Transcript

Daniel Kang: (0:00) So if I want to somehow prove that I'm human today, the way that I need to do that is essentially reveal a lot of information about myself because AI generation methods are becoming so powerful. Maybe we don't want this. So, for example, you have services which take pictures of your face to verify that you're a real human being. But the way this works is that they literally upload your biometrics to a server. There's a lot of potentially problematic things that come with this, including the potential for surveillance. But cryptography gives us a hammer that lets us bypass this fundamental trade off between privacy and authenticity. That's what I'm really excited about.

Nathan Labenz: (0:41) Hello, and welcome to the Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz joined by my cohost, Erik Torenberg. Hello and welcome back to the Cognitive Revolution. Today, I'm very excited to be speaking with professor Daniel Kang, assistant professor of computer science at the University of Illinois, who has done pioneering work bringing zero-knowledge cryptographic proofs to the domain of AI inference, making it possible for people to prove that a model has been faithfully executed as promised based on certain pre commitments and cryptographic calculations. Now as someone who knows very little about cryptography and the modern crypto space writ large, I have been wondering for some time what it might mean for AI to put the smart in crypto's so called smart contracts. As someone who has developed some very Byzantine heuristic systems in my day, it always seemed to me that the core limitation of smart contracts has been the fact that everything needs to be fully spelled out with explicit code. So could the use of language models in the context of contract execution and dispute resolution change the potential for smart contracts and perhaps a lot more besides? It's been surprisingly difficult to find someone who could inform my thinking on this question, but Daniel Kang is indeed the perfect person. Motivated by the growing trade off we face between authenticity and privacy, Daniel set out to develop methods that would allow people to answer key questions without having to reveal the specific inputs used to calculate the answer. Now that's a mouthful, so here's a couple of examples. 1, a human might prove their humanity by running an AI model on biometric data collected and analyzed locally on their phone without having to reveal the biometric information itself. Or an AI model provider like OpenAI might prove that it ran the model it promised to run and not a smaller, faster, cheaper 1 instead. And, yes, as in my original thought experiment, an AI might be used to resolve disputes where multiple parties can submit private data and all can verify that the model ran properly. Daniel even envisions broad use of so called attested devices, which cryptographically sign the data they capture immediately on device so that we can trace information such as photographs or audio recordings back to their physical source. The cost and complexity do remain limiting factors for these techniques for now, but Daniel does see a path to making such techniques a seamless part of our future AI infrastructure, much like HTTPS is for web traffic today. And the fact that he undertook this work in the first place shows an unusually forward thinking approach. So I was really excited to ask him not only all about how this technology works, but also about how he expects it to be used as society adapts to and hopefully takes full advantage of AI. To be honest, I'm not sure I ever achieved the level of intuition for the cryptographic math that I might have hoped, but I definitely came away with a much better sense of what sorts of things can be proved, why we might need such proofs in the near term future, and how this technology can be useful in all sorts of as yet undiscovered and likely quite unpredictable ways. As always, if you're finding value in our attempts to understand the near term future, we always appreciate your reviews and comments. 1 that recently came in on Apple Podcasts made my day. The reviewer says, I'm a full stack web developer with an interest in AI. I've followed generative AI loosely for a while, but with ChatGPT and all these amazing products coming out, I felt like I had severely fallen behind. So when I found this podcast from the very first episode, I was hooked. Each episode was not only informative but also thought provoking. I went back and listened to all of the podcast episodes from the start. And after each episode, I'd spend time learning about any concepts I didn't quite understand. I've used the podcast as a guide and, as of this week, a catalyst for my career growth. I have just accepted a position as a full time AI engineer at a Fortune 100 organization where I'll be helping other developers learn and use AI to build AI powered products. I can't wait to give back to the community. Thanks, Nathan, for being such a great scout through this cognitive revolution. All I can say to that is wow, and thank you. And I hope to continue to live up to that review, starting with this conversation about the application of zero-knowledge cryptographic proofs to AI inference with professor Daniel Kang. Daniel Kang, welcome to the Cognitive Revolution.

Daniel Kang: (5:39) Thank you. It's great to be here.

Nathan Labenz: (5:40) Yeah. Very excited to talk to you. This is a topic, that that which you are expert in that I know very little about, but have been kind of speculating about, wondering about, and frankly, trying to find the right person to talk to for a while. So I'm I'm glad that this day is finally here. And the the topic broadly is the intersection between AI and, quote, unquote, crypto, you know, maybe more specifically cryptography because this is not like we won't be launching any coins on today's, episode. And more specifically, you have done some really interesting work around the use of zero-knowledge proofs, within the context of machine learning. And that has some, I think, very interesting properties, and I think people will, you know, hopefully get a little bit better sense of just what you know, just how strange the future might be in some regards by understanding some of these possibilities and and the problems that you're trying to get ahead of, to solve. So I think it's gonna be super interesting. I definitely ask you to bear with us a little bit as we go because I know very, again, very little about the crypto space, and I expect that our listeners who are, you know, obviously very interested in AI probably also have some gaps in their knowledge around how some basic stuff works. So I did prepare by listening to a couple of previous, appearances that you've made where the audience and the hosts were more crypto savvy. Here, I think, you know, we've got a little more of the AI savvy, but the crypto is is probably, you know, highly varied. I'd love to just start off with articulation of what motivates this work, kind of what's the problem that, you know, you're anticipating and and wanting to solve. And then we can get into some of the mechanics of the zero-knowledge proofs and how that all works. And then kinda circling back toward the end, I wanna, you know, open it up a little bit and and envision, you know, a world where this stuff is really working and and kind of how it might ultimately be deployed. How's that sound?

Daniel Kang: (7:34) Yeah. Happy to talk about that.

Nathan Labenz: (7:35) What's the motivation for bringing zero-knowledge proofs to the AI space?

Daniel Kang: (7:40) Of course, the long term vision I've been thinking about is that as AI increases in its power, it seems like to me there's going to be this really fundamental trade off between privacy and authenticity. So if I want to somehow prove that I'm human today, the way that I need to do that is essentially reveal a lot of information about myself because AI generation methods are becoming so powerful. But I think that maybe we don't want this. So for example, today, I've actually heard dating apps are one of the biggest users of this. You have services which take pictures of your face to verify that you're a real human being. But the way this works is that they literally upload your biometrics to a server. And I think that there's a lot of potentially problematic things that come with this, including the potential for surveillance and other negative consequences. At some fundamental level, without cryptography, there's no way to get around this, as far as I know. But cryptography gives us a hammer that lets us bypass this fundamental trade off between privacy and authenticity. And that's what I'm really excited about.

Nathan Labenz: (8:57) Yeah. That's interesting. And and that's kind of another I think, you know, we can unpack a variety of use cases over time. This 1 has kind of become somewhat more well known in the last few months with the Worldcoin release. I think, again, most folks have probably heard of that and maybe have a sense that and this is about where my knowledge also stops that, like, the part of that scheme is that it allows you to prove that you are a human and, you know, tries to give kind of each human 1 unique ID to be able to prove that motivated there by the, you know, as you said, the fact that, you know, basically, since the this person does not exist, you know, .com website went online, a simple, you know, square headshot isn't gonna suffice anymore to prove that you're human, and so we need kind of ever escalating ways. And, obviously, those could become more and more sensitive. Another really interesting 1, just to kind of preview the breadth of application potentially for techniques like this, is basically what I'm starting to call model fraud. The notion that if you are paying for an AI model to be run in any, you know, number of contexts, how do you know that the thing was actually run in the way that you wanted it to be run, which could be as simple as just kind of, you know, a provider cutting costs, you know, and cutting corners potentially at your expense, or it could be, you know, all sorts of more nefarious attacks, you know, more more intentional malicious attacks than just laziness. So tell me a little bit about about that too because I know that's also part of the motivation.

Daniel Kang: (10:27) Yeah. That's another application I'm really excited about. So maybe to set the context a bit, if you look at what OpenAI does with their services, they will change the model that you're that you're being served, that you're that you might even be paying for without any warning and without any guarantees of quality. And so if you, in fact, look at some analyses over time, there's a great article on paper associated with that where some folks at Stanford and Berkeley measured ChatGPT's performance over time, and they showed it substantially degraded. So it went from something like, on a particular task, say 93% accuracy down to like 7% accuracy. And as a customer, you have absolutely no recourse today because you have no idea what OpenAI is running. You might have some suspicions, but there's currently no way to check if OpenAI is running a certain type of computation. In this case, a machine learning model. The same cryptography that allows you to prove authenticity can allow you to prove that OpenAI run a specific model, say the same model that was running a month ago, which could potentially allay some concerns about performance drops in, say, services like ChatGPT and other kinds of ML services as well.

Nathan Labenz: (11:36) So 1, short footnote on that for practitioners using the API. If you trust them, they do now have at least a bit better answer to that question than they've had in the past. They recently did introduce a, basically, a model dating scheme where, again, assuming you trust that they're doing what they say they're doing, they at least now say that they are freezing model weights, according to these time stamps and then, you know, giving developers a chance to kind of revalidate and, you know, upgrade at their convenience when new models come out. So contingent on trust, you know, with OpenAI, in particular, that's a little bit better than it used to be. In general, though, you know, the problem remains, right, that a, if I don't trust them, you know, then all bets are off. And, b, there's a lot of other providers out there, you know, increasingly proliferating, you know, which don't have such a a scheme and also probably have a lot less reputation, and a lot less reason, you know, to trust that they're consistent in their product delivery as opposed to OpenAI in the first place.

Daniel Kang: (12:37) I can also give another example, which is a bit more out there. So there's a lot of medical ML services that are proliferating. And so you could imagine one way to a malicious attacker, perhaps even a state actor, incapacitate someone is that if you send them in, someone goes in for a routine checkup, say, to check for cancer or something else. They can simply just replace whatever, say, ML model might be predicting to either flip the diagnosis in one way or another. So for example, someone does have cancer, you can report that they don't, and they won't get any treatment, and that they might suffer from the delayed treatment. But on the flip side, if they don't have cancer, you can simply flip the prediction and say, they do have cancer, and then they might get, say, treatment which could harm them despite not having any ailments. And so today, we have absolutely no way of telling whether or not a prediction from a medical model actually came from the model that was, say, certified by the FDA or not.

Nathan Labenz: (13:35) So that's a pretty good, starter set of use cases. And let let's, get into the guts of the technology a little bit more and then, you know, kind of map this back onto a little bit, you know, more of a futuristic vision toward the end. So here's kind of the headline result, and then I'm, you know, gonna rely on you much more to unpack it. But with these zero-knowledge proofs in general, the idea is that you can prove that a certain process was executed faithfully, but you can do that in a way that does not reveal the inputs to the process. And if you frame the machine learning inference problem as one of those processes, then the inputs become the weights of the model itself, which, again, obviously, is something that providers, you know, very much in many cases don't want to share and will choose to keep private over, you know, probably any number of other, goods that they might value, including, you know, the ability to verify in the absence of a technique like this. And then the other input is your input, you know, whatever you put in at runtime. And so the the technology allows you to kind of selectively expose and and different parties to know some of the inputs, but ultimately can prove this execution without revealing any information about the inputs or the intermediate states, just that the final result was a faithful execution. Is that right? That's right. Hey. We'll continue our interview in a moment after a word from our sponsors.

Daniel Kang: (15:10) The most salient thing here is that it forces the person or the party that's producing the proof, for example, OpenAI or a medical ML provider, to stick to that computational process. So not only do you prove that the computational process ran, but the prover cannot cheat. They cannot actually run some other process. Let's say that you want to prove that you ran on model A, they can't turn around and set what on model B, and then use that result instead. They're forced to use model a. So let's

Nathan Labenz: (15:40) break this down a little bit at maybe two different levels. 1 is kind of the protocol, which is, you know, basically crypto language for the algorithm of execution. Right? The steps that the different parties have to take in order for this to you know, for the guarantees to hold. And then going a little deeper than that, I'd love to try to understand the math and at least get some intuition for why we can believe this, you know, at the most fundamental level. But let take me kinda through the steps that I and OpenAI would go through if I wanted to you know, and and they were willing to make these sorts of guarantees. Like, who has to do what in what order?

Daniel Kang: (16:24) Yeah. I think it's worth, drawing an analogy to how you would take a machine learning model and deploy it on a GPU. And in some sense, you can think of the zero-knowledge proof as a GPU that proves that I ran a specific model. So if you want to deploy a model on a regular GPU, what you first have to do is specify the model architecture. Common tools are like TensorFlow and PyTorch for this. So the framework that I've built accepts currently variants of TensorFlow and a format that PyTorch exports to. And then you specify the model weights in addition to the architecture. You specify the input, and then you say, run this on the GPU. Similarly today, that's what the stuff that happens. You first start off by specifying an architecture, the weights, and the input, and then you pass it to the system, which then proves that the model executed correctly. And so from the perspective of both the model provider and also the person who's receiving the proof, it's all fairly black boxed. You can simply run a program so the model provider can produce a proof that says, I ran this specific model on this specific input. And the person who's receiving the proof and the prediction can verify that that happened honestly.

Nathan Labenz: (17:36) There are some additional time sensitive sequences, in that. Right? Like, for me as a customer to ultimately have confidence in the final assertion, If I understand correctly, the provider needs to say, okay. First of all, here is essentially a hash of the architecture and the weights. Like, there's a bundle of sort of what defines the model that gets kind of hashed out to a alphanumeric string or whatever, right, that I can sort of say, okay. This was and that's basically a precommitment. They have said, you know and this this thing is inalterable. Right? So that I have to have that in advance. Then I can call the model. They can execute. Then they can also produce the proof. Now I have both the precommitment artifact and the final proof, and then I get to do something on my end to verify. Right? With those based on those 2 things, I can sort of verify that the whole thing was legit end to end. Is that right?

Daniel Kang: (18:42) Yeah. That's right.

Nathan Labenz: (18:43) Okay. Cool. So help us understand the you know, how this is working under the hood. I think your GPU analogy is really interesting. I, in general, you know, listeners of the show will know I'm always really reluctant to base my understanding on analogies. I think that, you know, obviously very useful in many ways for developing intuition, but I try to hold myself to the standard of I wanna understand this. I'm happy to, you know, to simplify some details, but I kinda wanna find a way to understand it that is, like, very literal even if simplified, you know, and still not wrong. That's kind of the base camp for future understanding in my view. So help us kinda get to that. Like, what is going on at each of these steps when, first of all, you know, a precommitment is made, and then a proof is made, and then a verification is made? All 3 of those things have to happen. And if I understand correctly, they the the model itself, you know, that that bundle, that's a one off process. So they can basically publish almost you know, plays a similar role to, like, a public key, I guess. That's like, this is a starting point. Then for each instance that we wanna prove. Right? I don't know I don't know if it would come down to each token or each you know, if it could be something that's, like, for a full, you know, long text generation. But we have to kind of do this proving and verifying each time. Right?

Daniel Kang: (20:04) Yeah. So for every input, you have to produce a new proof. That's correct. So so in terms of how it works, cryptography is one of these things where there's many levels of abstractions. And so I'll try to aim at one level of abstraction. You can tell me if I'm going too deep or too shallow. The commitment is the easiest thing to explain, I think, because people have actually most people have used this in some form or another. The commitment is simply a hash. So a cryptographic hash is it's literally a function, a program that you've run, and you assume that it's really difficult to produce an input that produces the same hash as the 1 that you produced. So in general, if I produce a hash, I know that a program actually runs that hash, then I can essentially prove that I know some inputs that that resulted in this hash. So this is actually what secures the Bitcoin network and billions of dollars in value. So hopefully, fingers crossed, that hashes stand the test of time.

Nathan Labenz: (21:03) And just for a little bit more color there, these have been around before crypto. Right? Like, I've I have used hash functions in programming even for just, like, you know, hacky things like uniqueness checks because of the property you said. Right? I mean, there are a lot of different hash functions out there. They generally have this sort of I don't know if it's proven or if it's just, you know, you you said kind of assumptions, so that's an interesting wrinkle. But, like, in general, it's like there are, you know, more possible hash outputs than, like, atoms in the universe or whatever. And, you know, there's some mapping from large inputs to these relatively small, presumably unique outputs such that it's, like, pretty commonplace in just general programming to use them to dedup larger objects to, you know, have have kind of a database of hashes where it's like, okay. We wanna check, you know, is somebody uploading some copyrighted material straight away, like, to just hash it and and to kind of check. So that's fairly familiar, but I I am interested in that wrinkle of what are we assuming there? And, like, what just just the sanity check. Like, would we would this be like a quantum, you know, computing future that might break that assumption perhaps?

Daniel Kang: (22:11) So there are many different kinds of hash functions, and different hash functions have different assumptions. So hash functions are one of these things where they fall under the broad umbrella of cryptography, and typically what you do is you assume some property. So a common property to assume cryptography in general is that inverting a logarithm in a finite field is hard. And so if you assume that, then you can assume that certain kinds of computations are difficult. We also have some hash functions which are assumed to be quantum resistant, But it's one of these things where there's no strict mathematical proof that these things are impossible. But they're widely used in practice. And as far as we know, after decades and decades of research, there's been no actual, basically, ways to break these hash functions.

Nathan Labenz: (22:58) So okay. We've got, no formal guarantees, but we do have decades of practice. Are there, like, examples of hash collisions, you know, with kind of modern hashes where people report things going weird on them, or is this just, like, totally smooth sailing in practice?

Daniel Kang: (23:16) Yeah. That's a that's a complicated question to answer because there's two levels to this. one level is that if you think about passwords, there's actually whole engineering teams at the dark corners of the internet dedicated to reverse engineering passwords. But that's because passwords have essentially limited entropy. So most like, the most common password is Password. And so, you know, like, if I just look at a random password and I just guess, okay, that's the input's probably password, then I get it with, like, 10% probability or something. But if I'm trying to be secure about this, and I can add let's say I choose a truly random password, I do what's called I salt the password, then it's assumed to be extremely difficult. Requires a lot of computing power to to break that.

Nathan Labenz: (24:02) I've used this technique a little bit, but, basically, the salting is you take the original raw input, you add some kind of secret additional input, and then hash that.

Daniel Kang: (24:13) Yeah. So secret additional input also adds entropy. So it becomes hard to guess because it's it's random. Or if you're doing it correctly, it's random. So I I describe this in the context of passwords because passwords are the most common way people interact with hash functions. But in the case of trying to protect your secrets from an ML model, what you can basically do is you can solve the ML model. You take the ML model weights, which might have low entropy, and you add however many bits of entropy at the end. And then by doing this, like, because the last set of bits are random, the hash becomes essentially random.

Nathan Labenz: (24:48) We have a trained model in this scenario where we want to prove the model ran without really revealing anything about the weights. We first can take the trained model, represent it in some form. Well, there is still a trick here. Right? Because I am not going to ever have access presumably to the the model to, like, rerun the hash, but yet that hash is now gonna feed into this scheme to allow me to prove something downstream of it. So that's interesting. So, okay, let's keep going. We've got the thing hashed. Now we come to the point where we want to prove that the computation ran. So describe that proving process.

Daniel Kang: (25:28) Okay. So at the highest level, the technology we're using is called a zero-knowledge proof, and we're using a specific form of a zero-knowledge proof called the zkSNARK. The zkSNARK, or sorry, zero-knowledge proofs in general, allow you to prove that any arbitrary computation ran correctly. The key here is to choose what computation or what function to prove. The function that we're going to prove has two parts. So the inputs are the model weights and the hidden input, for example, user prompt. So when we have this zero-knowledge proof, we can reveal any part of the computation. So we might reveal the input as a concrete example. Now, inside this function, the first thing we compute is the hash of the model, along with the salt. And then we reveal that hash. So because I computed the hash inside the zero-knowledge proof, I, as a model provider, cannot use a different set of weights, because the hash you can think of as binding and the zero-knowledge proof is also binding. Then the other thing you prove is that you actually ran the ML model, the architecture correctly with a given weights of input, and then you can reveal the output. And so by doing this, you essentially forced a fixed architecture within the function specification. You force a fixed set of weights by revealing the the salted hash. And then you've also forced the input by revealing it.

Nathan Labenz: (26:51) Okay. So let's dig in a layer deeper then. How it's very, unfamiliar territory for me, obviously, but what I found kind of interesting about this is models themselves, right, are all linear algebra, all matmuls. That's not quite true because there are also these nonlinear layers that sit between, and it does seem to be a you know, I don't know if this is quite proven either, but it does seem to be a important and pretty robust empirical result that the best results kind of come from this mix of lots of linear algebra followed by this kind of nonlinearity that acts seemingly as a sort of filtering function to kind of allow certain meaningful signals through and, you know, sort of stop other signals that are just noise. Something like that seems to be going on. Then in the context of this proof, there's kind of a process of arithmetization, which is taking this computation, right, which is, as you mentioned, kind of ultimately defined by an architecture, executed with, you know, a specific set of weights on a GPU. A lot of arithmetic there, but not entirely arithmetic given the nonlinearity. And, essentially, the first big part of this is creating an arithmetic approximation of that original kind of, you know, proper computation. First of all, do I have that right? And second, like, I'm still not quite sure, like, why are we making an arithmetic approximation in the first place?

Daniel Kang: (28:23) Okay. So once we get to the point where we have a zero-knowledge proof, so ignoring ML models entirely, just if you have an arbitrary function like you wanna prove. Our optimization refers to the process of taking a function which might be specified as, say, a Python program, and turning that into the format that the proving system, the thing that produces the zero-knowledge proof, can actually ingest because it doesn't take in Python programs. I believe the reason it's called authorization is that historically, zero-knowledge proofs were constructed using this thing, this mathematical object called an arithmetic circuit. So you have to represent your function as an arithmetic circuit, so you arithmetize the function to the arithmetic circuit. For the zero-knowledge proofs that we're talking about for ML models, the arithmetization is simply taking the ML model architecture and weights and then turning that into a format that the zero-knowledge proof can ingest. At this point, there's actually no approximation. It's a it's an exact one-to-one mapping.

Nathan Labenz: (29:24) Okay. We were doing a ton of arithmetic plus these nonlinearities, and now we're essentially a state restating a ton of arithmetic as arithmetic. And then you have kind of an additional element where you're using lookup tables for the nonlinearities to basically say, if I understand correctly, because the proving system doesn't accept that kind of function as an input that it can do proofs over, we need a I don't know if I would call it simpler, but we need a, you know, more explicit numerical representation of the nonlinearity. So I guess there's basically, like, equivalent of, you know, those sort of renaissance, log tables, you know, where all of these values are just kind of precomputed and then can be looked up. So is that the main transformation that's happening in the first place? Like, it's it's all exact in the arithmetic, and you have, like, high precision on these lookup tables. And so do I have an artifact there that gets input into the moving system that basically could be run in place of the model? Like, it's it's truly a like for like. Like, it will it would generate the same outputs?

Daniel Kang: (30:43) Yeah. That it would generate the same outputs. So you do have to quantize the model, which is actually a very common thing to do with machine learning models in production. OpenAI does this and a bunch of other, pretty much all other serving systems do this in some form or another. So once you do quantization, it is an exact representation of what happens. Everything is correct from an intuition standpoint. I guess one thing I might make a little bit more precise is that the lookup tables are also exact results. So for example, if you have a quantized input and then you apply, say, a ReLU or a tanh nonlinearity, you can just get the exact output after applying a lookup.

Nathan Labenz: (31:20) And so literally, there's a value for every I think the was the phrase in the paper exhaustively, precomputed, which basically means, like, down to the level of precision that is possible every last if it's 8 digits, you've got 8 digits worth of lookup values waiting to be referred to. Fascinating. Okay. Can we take just a little bit more of a detour into the floating point versus the fixed point operations? I think this is something that, honestly, most people probably totally gloss over. I think what we know what I feel like I know is, okay. Naturally, if I've got, you know, whatever, a 64, you know, digit number versus a 32 versus 16 versus an 8 versus a 4 intuitively makes a lot of sense, then, like, I would need less memory. You know, it would be faster to do these whatever compute process over just smaller numbers. That would be true if I was doing it by hand. It seems, you know, intuitive that it'd be true doing it on a computer. But it seems there's a little bit more kind of stuff going on there than just that there's something more fundamental where this change has to be made. And it's it's, like, probably in practice fine because it's often made for these efficiency reasons. But it seems like this is not an efficiency question in this context. It's more of a, like, validity question.

Daniel Kang: (32:42) So zero-knowledge proofs can let allow you to prove arbitrary functions, so you can prove the full floating point arithmetic if you want. It just becomes much slower to produce the proofs.

Nathan Labenz: (32:51) Gotcha. Okay. So it's the same basic concept. Is it just computationally saving to go with the quantized version?

Daniel Kang: (32:59) Yeah. Yeah. And quantization is used very widely. I believe, recently, Elon Musk, revealed that they use quantized models within the Tesla vehicles. So you have millions of cars running around on the roads, with these quantized models, and they're used by by Google, OpenAI, many other services as well.

Nathan Labenz: (33:18) So we're up to the point now where we have having precommitted, we now have a fully arithmetic representation of the computation that the model's actually going to perform, including this main change being the substitution from a traditional nonlinear calculation to the exhaustive lookup function that has 10 to the 8 things in it. Right? Or 8 to the 10, I should say, maybe.

Daniel Kang: (33:49) So because we do quantization, we actually don't need to represent that many digits. Like, it's more like something like 2 to the 15 or something. So, like, you know, on the order of, like, hundreds of thousands or maybe slow millions of possible inputs.

Nathan Labenz: (34:02) Okay. So now this all gets ingested into the proving system. And I guess this is where, you know, it probably starts to get a little bit crazy. So what happens next?

Daniel Kang: (34:14) Yeah. So there are many different kinds of perfect systems for zero-knowledge proofs, and there are many different kinds of zkSNARKs. Talking about all of them in this full generality is little bit outside the scope of this podcast, but I can talk about the general intuition. Well, actually, is somewhat precise. So almost every proving system I'm aware of relies on the fact that polynomials can represent arbitrary computations. So if you allow the number of variables in a polynomial to grow, you can literally represent any computation. So maybe some professors will get mad at me, but I think you can basically think of this as if you grow the number of variables in a polynomial, you can write down anything you want within that polynomial.

Nathan Labenz: (34:57) This is basically like Taylor expansion like work. Right? I can recall doing these sort of, okay. What's the first, you know, approximation? What's the second approximation as I, you know, enable a square parameter and a cube parameter? And, you know, lo and behold, you can reconstruct the sine wave with if you allow sort of infinite terms in that sequence. Obviously, you know, you don't have to look through infinite terms, but that's basically the same type of math. Is that right?

Daniel Kang: (35:28) So the intuition is the same. But because we're working over finite fields, I think maybe a more precise analogy might be sorry, if you have 2 logic gates, a NAND and a NOR, you can literally represent any computation. And then 1 reduction you can do is that you can actually if you have a polynomial of multiple variables, you can literally just take the variables and then multiply them and add them in certain ways, literally represent NAND and NOR. And so then the polynomial gets reduced to these NAND and NOR gates, and then the NAND and NOR gates are then universal in terms of any computation. So a reduction to polynomial can represent any computation.

Nathan Labenz: (36:02) Yeah. Can you unpack the finite fields bit a a bit more? This is kind of where the we make the leap into cryptography. Right? Because the finite fields basically has to do with using prime numbers in some way to basically make the the math hard to reverse, right, or hard hard to kind of tell me sharpen me up.

Daniel Kang: (36:26) So finite fields, you can basically think of as integers from 0 to some large number. And then once you get to that large number, you wrap around back to 0. That's the basic intuition behind the finite field. And then you have certain operations. The operations we care about are plus and minus multiplication. And then there are constants, which are from 0 to the large prime number that we care about. So 1 other thing is that we actually also operate over elliptic curves at some point, but that's maybe not super relevant for the purpose of this discussion. The reason that we use finite fields is that polynomials or finite fields have this amazing property, which is that if you have two polynomials, let's call them f and g, and you take a random number in the finite field, and you compute f of the r, let's say, the random number, and g of r, the random number, f and g, unless they're equal to each other, are not going to be the same at this random point. Like, the probability that they match at this random point is vanishingly small. So this lets you do something amazing, which is that maybe an easy way to think about this, so let's say I have F and you have G. And let's say that we're both honest for the time being. I can compute if let's say and we want to check that F and G are equal. Instead of sending a representation of f, which are, for example, coefficients, what I can simply do is take a random point. We both agree on a random point, and we just send f of r and g of r to each other, and we can check that the evaluations match. So the coefficients even differ at 1 point at 1 like one of the coefficients, even by off by 1, the values will be completely different. Okay, so taking a step back, as long as there's some way to force a prover to what's called commit to a polynomial, which basically says, I'm going to use this specific polynomial, then we know that they actually use that specific polynomial because of this amazing property of polynomials and finite fields.

Nathan Labenz: (38:21) Just zooming out. I I always kind of, you know, try to zoom out for a second and then zoom back into the part where it's confusing. So we've got our, you know, three-step protocol. We're going to commit with a hash to this is the model that we're gonna use. Now we're in part 2, which is we're going about the proving, and then part 3 is gonna be the verifying. In part 2, we first do an arithmetization of all the computation. This is an exact representation, but in a different form making use of, for example, lookup tables such that with that transformation, we could still do the exact original yeah. We could run that as our forward pass. But then that whole process gets kind of mapped onto this sort of wonky arithmetic that is operating over finite fields, which is, like, somehow all this is now represented in, like, modular math with a given kind of prime key. And the same sort of hash like property is preserved such that can only get to that result if I used the the same input that I promised?

Daniel Kang: (39:35) Yeah. That's more or correct. So the process of arithmetization is taking the computation, and you can think of it as turning it into a polynomial over a finite field. So polynomial is a representation of the computation.

Nathan Labenz: (39:48) And the whole secret here is that there's some private key prime number or large prime number that is being used that nobody knows. Right? And that's why, like, the whole thing becomes irreversible. And that's also the whole reason that we're using this prime based scheme in the first place.

Daniel Kang: (40:05) So for all the programming systems that rely on these these polynomials, in a specific way I'll talk about relies on what's called toxic waste, which is the term of art for this random prime so this random value that is used to ensure the secrecy because we can't reverse that random number.

Nathan Labenz: (40:25) So at some point, there was a couple of things that kind of got a little confusing here. So, you know, in in kind of zooming out and looking at the results, one thing that's interesting is the setup or that original kind of commitment, right, has a has a certain amount of compute attached to it. Then the proving also has a certain amount of compute attached to it. And then the verifying is, like, much faster. So it definitely makes sense to me given what we're talking. I mean, inference is, you know, compute intensive in the first place. So it's no surprise that doing this transformation and actually executing it takes a significant amount of time. But one thing that kind of was confusing to me is there at some point, there is a an approximation is introduced, right, or a a sort of lack of, like, full fidelity to the original. Like, at some point, I can no longer use this as a substitute for my original forward pass. Right? So when and how does that, complication arise?

Daniel Kang: (41:29) So that complication arises if you do quantization in a way that's not efficient and in a way that's, results in basically, bad approximations. We've actually fixed that in, some upcoming work. And so basically, you can think of it as a fully, faithful representation of the forward pass.

Nathan Labenz: (41:48) Interesting. Okay. That's it. That's a meaningful update, it sounds like. It's a matter of making a better quantization in the first place that gets you out of this indeterminacy.

Daniel Kang: (42:01) So there's there's no indeterminacy anywhere?

Nathan Labenz: (42:04) Yeah. That's that's loose loose speak. But I guess just let's back up for 1 second before we talk about the solution because in the original paper, which I was trying to make as much sense of as I could coming into this, there is a probabilistic sense of you can be so confident that the thing has run. And to kind of build more confidence, you have to, like, do more compute. Right? So there seem to be kind of a Pareto frontier there somehow where depending on how confident I wanted to be, I can, you know, invest more in, you know, a more rigorous kind of convincing proof?

Daniel Kang: (42:40) I see. I see. Yeah. So there's so there's actually two levels here. So 1 is the individual proof. The individual proof you can basically think of as unforgeable as an exact representation of what's going on. Once you have this primitive of a proof, then you can layer things on top of that. So for example, let's say I want to approximate GPT-4's accuracy on some task. Well, maybe I have a test set of 100,000 examples. Producing 100,00zero-knowledge proofs is very, very costly. So instead, what I can do is I can ask OpenAI, well, produce proofs on 100 of my test set. And I'm going to use that as a rough approximation of a measure of accuracy. And then 100 might be too small. I might instead ask for 1,000 or 10,000. But that is something that you layer on top of these proofs. The proofs themselves are you can think of as unforgeable, no way to cheat that. And then this test set accuracy computation is something you layer on top of these proofs.

Nathan Labenz: (43:40) Okay. So I think I misunderstood in a pretty important way what some of these percentages meant. Big update in my understanding here. So the individual proofs are fully robust. If I run this process, I, a, will get approved that a competition was run, and it will be sort of have all the nice properties of it. It'll be unforgeable. I can fully kind of, you know, be confident in that. And then the down the the subsequent kind of layer is okay. But this is still expensive. Right? I mean, it's essentially a another forward pass, but even more computationally intensive. Right? Because we've gotta do all this kind of weird math that does it in the, you know, the finite field. So it's even more, you know, relative to running your forward pass. This kind of provable running of the forward pass is just a lot more expensive. Is there a ratio? Should we have a rule of thumb in mind for how much more expensive?

Daniel Kang: (44:38) With the work that we did in the paper, can think of it as something like 10,000 times more expensive, which is quite quite out there. Fingers crossed in the next few months, we'll have some work that bring down this cost.

Nathan Labenz: (44:48) So orders of magnitude more expensive. Hopefully, orders of the number of orders of magnitude coming down. But obviously, that is a lot. So, you know, basically, then if I were gonna do 10,000 inferences and I just wanted to prove 1, I would be doubling my total compute bill.

Daniel Kang: (45:05) That yeah. That's roughly right.

Nathan Labenz: (45:07) And obviously, again, you're you're working to make that more efficient. So then naturally, it follows from that that I wouldn't want to verify every single 1 would be infeasible. So I have kind of more statistical classical statistical sampling techniques that you're just kind of mapping out for people saying, you know, okay. Here's you know, you can do this many on a set of this many, and you can kind of get this confidence. Okay. That all, I have much more intuition for. Now I think we're basically getting to the end of part 2, right, where we've created a fully faithful and unforgeable, albeit 10 10000 times, you know, or or hopefully soon less, more expensive version of the same computation that'll that has all these nice properties for verification. And then what is the proof of itself? At the end of this, it it kind of just spits out some, like, bit string, basically, right, that that is kind of the final proof and that's independent like, that that's sort of in addition to the actual output?

Daniel Kang: (46:13) Yes. You have to specify the proof and you also have to specify the output. You have to send both.

Nathan Labenz: (46:18) So then I get as a user, as a customer of whatever the model provider that I want to double check, they give me both things. I look at the output. And then what do I do to do the verification? I'm also running relevant additional software on my end. Right?

Daniel Kang: (46:35) You can think of this as verifying a hash on your end. You basically kind of how you send checksums or hashes with large files to ensure that it's got sent over the wire properly. You can sort of and you run the hash on your end as a user. You can sort of think of this proof as like a hash, quote unquote, of the computation, and then you just verify that it matches. And to do this, you do some more basically finite filter.

Nathan Labenz: (47:03) But it's way easier this time because I mean, this is just one of many things that it's, like, much easier to verify than to do. Is there any more intuition I should have for that?

Daniel Kang: (47:15) It goes back to this thing where you have to basically check polynomial equalities. As the prover, I basically need to say, okay, I'm actually gonna use all of these coefficients and this fixed polynomial, which requires a lot of effort. But then as a verifier, you can just say, okay, well, if I know what polynomial form is, I can just check it at a random point. And so I have do much less work than the proof does.

Nathan Labenz: (47:36) So I'm getting the polynomial form or that that is the the proof represents the polynomial form? Because, like, because the polynomial form is irreversible, I can actually receive the polynomial form without being able to run the model on my own downstream?

Daniel Kang: (47:56) Well, I'm using this term colloquially, but it's called the verification key, which specifies the layout of the computation, which in this case is basically what polynomial you're using. And then you can, like, basically just check that that polynomial was run correctly.

Nathan Labenz: (48:10) If we do set this up and I am the the client, the verifier, and I get this polynomial form, does that mean that I basically have a 10,000 times slower instantiation of the model? Could I could I somehow run that with other inputs if I were willing to do it at 10,000 times slower, or do I still not have enough information to perform, like, inference? You could do it

Daniel Kang: (48:40) with your own weights, but the the weights are will not be revealed to you. So basically, polynomial encodes model architecture, but not the weights. So the polynomial represents the computation of compute hash and compute model architecture with a hidden set of weights. But because the weights are hidden, if I send you this polynomial, you can run the polynomial on your own, but you won't have my weights.

Nathan Labenz: (49:09) So is that it's dependent on my specific inputs? Like, I'm just a little confused as to how how we somehow make this jump from there's a fully faithful reformulation of the computation that I can verify but still can't run it on inputs in general. Right? Like, I I can run it on my input and confirm that it works, but I can't run it on arbitrary inputs. How does that how does that narrowing happen?

Daniel Kang: (49:40) Here's an analogy. So Twitter recently open sourced their recommendation algorithm. But even though the code is all there, you can literally just look at the PyTorch architecture, you cannot run that code like as Twitter does, because Twitter has information that you do not have. So you can think of the verification key, this polynomial form, as the PyTorch architecture. But even though you have that PyTorch architecture, you still cannot run Twitter's recommendation algorithm. So so if I send you a Python file which says, okay, so here's the Twitter architecture. We have some multilayer perceptron with some skip connections and all this kind of other stuff. And then you ask them, okay, so giving this giving this PyTorch architecture, can you actually, like, verify that the timeline is generated from this architecture? The answer is no. There's no way because Twitter has the input data and the the model weights, which are unknown to you.

Nathan Labenz: (50:38) But in our main case, there's I'm trying to isolate this this 1 kind of leap that is that I'm not quite following on where I mean, it it depends on the initial commitment. I I understand that. Right? There's some, commitment to particular weights, and that is needed I guess there's, like, a, you know, some sort of factorization or whatever happening at the end that is basically saying, because I know this hash of the weights, I can do some sort of thing, but that also depends the final verification depends on my input. Right? So I I couldn't verify for any other input. I'm not quite getting why that is. Because that seems this you know, in the case of the Twitter thing, I'm basically just saying, well, yeah, we haven't given but they also haven't given me the ability to verify. Right? So I I can't run it, but I also can't verify it. Here, I'm getting this ability to verify it, but I'm trying to figure out how the ability where exactly in the in the steps the verification comes through and the ability to, like, redo that on a different input stops or, you know, doesn't come with the ability to verify.

Daniel Kang: (51:50) Okay. So this polynomial form that we've been talking about, let's call it the verification key. You can think of the verification key as the model architecture plus the hash. That's the the the right way to think about it. So if I have this verification key, I can put in my own weights and then run the Twitter algorithm. But the hash is not going to match because I don't know the weights as a verifier. So even though I have this verification key, I cannot produce a proof that matches with this hash because I don't know the weights. Only the model provider who knows the weights, for example, Twitter, can produce this, like a passing proof, because when you plug in the weights, it will match the commitment. That's what forces Twitter to use a specific model, a specific set of model weights.

Nathan Labenz: (52:44) Yeah. I get the scheme. I still don't fully get the math, but I do get the scheme, and I'm probably not gonna get the math today in its full complexity. So might be time to give up on that dream. But so just to to kind of restate the scheme, the step 1 is the commitment to the hash that defines the architecture. So there's some transparency around the architecture, but the weights themselves are are just in this, you know, kind of compressed hash With the process of the arithmetization and the, you know, 10,000 x, hopefully soon lower, increase in compute cost, I can do this math over this strange kind of large prime modular space instead of in its original arithmetic form. That's a lot more work, but it also allows me to kick out at the end a another basically hash looking thing that constitutes the proof. And then the prover at the end as the verifier, what am I running at the end to do the check?

Daniel Kang: (53:46) You're basically evaluating polynomial to run a point.

Nathan Labenz: (53:49) So I'm confirming that the thing that I got back is producible based on the previous commitment that I got and the inputs that I personally already had that were known to me.

Daniel Kang: (54:04) Yeah. So you're basically checking that the hash, the commitment of the model weights match, the architecture matches, and the input matches. That's what you're that's what you're doing into it.

Nathan Labenz: (54:12) I think this is definitely super fascinating. I mean, the to zoom out. Right? The we're entering a world where AI agents are, you know, getting worked on feverishly, and people are kind of looking at a future of all sorts of autonomous things. People are gonna need to prove that they are human. You know, agents or models are gonna have to prove that they kind of ran the right thing. And so all this stuff can kind of be used anywhere where the value and and reliable even with kind of a single, and this is the big probably biggest update in this conversation for me is that it's not just a probabilistic thing, but it is a fully guaranteed thing. So I could do this just you and me 1 on 1. You have a model. I want you to run it for me. I don't need any other parties. I don't need any other reputation mechanism or platform. If we're willing to pay the compute tax to do the verification, then we can verify that you're running what you said you were going to run.

Daniel Kang: (55:12) Yeah. There's there's only third parties. You can literally just do this on your own, and there's no way to cheat it.

Nathan Labenz: (55:18) Let's talk about what this enables. I mean, one thing that I've been looking out for for a long time, I mean, long time, whatever, long time in AI years, which is, like, 1 year. But, obviously, so much crypto infrastructure has been built up and there's been this promise of smart contracts. And when I look at smart contracts, I'm always like, well, they're typically limited by the limitations on the smart. Right? I mean, what we've had historically is, like, auto executing contracts, which has been labeled smart contracts, but, you know, they're still kind of traditionally really constrained by just the fact that, like, they're running explicit code, and explicit code is hard to make smart, especially around, like, real world, you know, complexity use cases. Certainly, when it comes to disputes. Right? I mean, people are there's, you know, there's a lot of dispute resolution mechanisms kind of proposed for crypto. But in the end, it's like as long as you're relying on explicit code, it's it's a fairly brittle thing. And, you know, I might prefer to have recourse to a human. Now it does seem like there's this opportunity to start to use language models to help do some of this, you know, kind of on chain, you know, trustless arbitration, you know, dispute resolution of all sorts. And this goes back to my I was a member of the GPT-4 red team. one of the things that I tried in that early testing was this wasn't even really a renting thing. Was just like, was very interested in this. Could we get GPT-4 to do basic mediation and dispute resolution? And sure enough, like, it's quite good at that, especially as you look forward into a world of like multimodality, you know, and other kinds of evidence being able to be brought to bear besides just, like, the testimony of different parties, then it really starts to look promising that you could have first line dispute resolution powered by AI models. But then it really starts to matter, you know, what model are we running and, you know, can we trust that and and who's doing this execution? So, you know, one way to do it would obviously be to trust a a third party. Even then, though, you know, if the stakes are high enough, I might want to audit, you know, a somewhat trusted, third party. I'd certainly wanna have some, you know, ability to do that in in cases where it felt necessary. It seems like that is is kind of coming online. Like, why why isn't that already here? Is it because this kind of work that you're doing has been the bottleneck to date? How do you I would love to hear more of your vision for the future of, you know, the the additional roles that, you know, language models and and multimodal language models can play now that we have this, additional layer of guarantees.

Daniel Kang: (58:15) Yeah. I think that would be really cool. There's a few challenges though. So 1 challenge is just the computation. So right now, if you think how much GPT-4 costs, you blow that cost up by 10,000, then

Nathan Labenz: (58:26) We're back to the cost of a human arbitrator.

Daniel Kang: (58:28) You're back to the cost of a human arbitrator. I mean, yeah, maybe a little bit under, but you know, still it's very, very expensive. I think the second thing, though, is that both parties have to agree to a specific model and agree to the outcome. I think this gets quite challenging because the specific way you phrase a prompt can result in different kinds of outputs. And there's a lot of human complexities when it comes to things like contracts. I personally think that liars are going to be the last people to be automated away because people don't like feeling like there's just a random machine making decisions for your life. That's just a personal thing. I think technologically speaking, the main blocker is just the cost.

Nathan Labenz: (59:14) So yeah. I mean, I think that's very interesting question to debate and speculate about. How much do you think the cost comes down? Because if GPT-4 is, let's say, you know, it's whatever. It's 3¢ per thousand input tokens, 6¢ per thousand output tokens, presumably that you know, they've got a long track record of reducing their prices, and that's obviously downstream of ever increasing computational efficiency on their end from all sorts of different, you know, optimizations. But okay. Let's say 4¢, you know, on average for per thousand tokens. If we do a 10,000 x on that, you're now looking at, like, $400, which, you know, again, is kind of order of magnitude human powered arbitration. Although, yeah, probably still a little cheaper, although maybe we need more than 1000 tokens too depending on the situation. How much do you think you can bring that down? Because it seems like the difference between 10,000 x and even a 100 x is, like, the difference between human arbitration level and, you know, couple percent of human arbitration level, which might be the whole game, you know, in terms of getting into a cost regime where you know, maybe it's not that the lawyers are automated away, but just that a whole new set of kind of protections and interactions are made possible that just previously you know, there wasn't enough recourse to create the the conditions for people to transact in all sorts of different ways.

Daniel Kang: (1:00:47) That's a great question. A lot of this is active research, so it's very difficult to predict where the final numbers will be. We do have some results, which we'll be releasing in a few months, which bring down the cost of certain kinds of models pretty dramatically. So think something like 10 to a 100 times lower cost. But transformers, so GPT-3, 4 style models are quite challenging due to the way the competition is laid out. So the cost savings there are unfortunately not quite as high. But this is also something we're actively looking into. And there's basically new developments, both on the cryptography side and also on just how you actually do this polynomial computation more efficiently side as well. Things are changing literally every couple of weeks in this space. So sometimes it's hard to keep track of It's hard to guess where the final numbers will be. But hopefully in maybe a couple of years, the numbers will be maybe a 100 times more expensive.

Nathan Labenz: (1:01:47) Yeah. And that does seem like it would be really useful, you know, just in my own day to day commercial life. Right? It's like, I've got an old house and it has, you know, these kind of idiosyncratic projects that it needs. And it's tough to engage, like, an individual contractor, you know, who might be good, might not. It's hard, you know, it's hard for them to develop a reputation. It's hard for us to kind of agree on, like, should I pay this whole thing up front? You know, you may ghost me. You know, there's there's all these kinds of friction points. If I had some ability to, you know, enable kind of order of magnitude $10 recourse, I do think that would be pretty interesting and and a huge enabler of all sorts of kind of more decentralized peer to peer commercial interactions. So I do find that, like, super interesting, super promising. A couple of the things that you've talked about in in some of your other works that I think are really interesting here too is the kind of secure provenance of assets. So, you know, like, if I was gonna hire somebody to, you know, fix my plaster wall in a, you know, idiosyncratic way, I would probably want, like, images, And I would want those images to be reliably known, like, when they were taken, you know, and maybe on what device they were taken. Sounds like this is actually quite a bit farther along than probably most people realize. So could you give us a little bit of and this isn't exactly necessarily your work, but I'd love to just kind of ground, you know, this future possibility and also the fact that we have, sounds like, pretty decent solutions for that kind of problem already too.

Daniel Kang: (1:03:32) Yeah. This is one of these things which I am personally very passionate about, but somehow I'm trying to, you spread the word as much as possible. So given the rise of AI generative images, with existing technology, you just cannot tell if something is real or fake. And the way to sidestep this is also cryptographic techniques, specifically what are called attested sensors for images and a tested camera. I also think that this is the way to get around these things where you have to go to an orb somewhere and then trust some company that they're not going to sell your iris information, which is apparently happening today. So the way that an attested camera works is that you have a camera sensor, which basically takes in light and then up emits an electric signal, which is basically bits. And what you do is you put a hardware device physically right next to the camera where the bits get read off. And then before those bits go anywhere else on your computer or anywhere else on the device, it essentially hashes the inputs and computes a signature of the hash. And so it's very difficult to forge these kinds of things. And so you can basically show that, oh, I took an actual image of, say, my wall in this plaster. Now, there are also some mechanisms which I think are very interesting, which can be combined in a way that you can also Approve is not the right word here. People will be mad if I use the word probe, but you can be convinced that someone took an image at a particular time as well. And by combining these along with, say, machine learning models, you can then do things, as you mentioned, like dispute mechanisms. You promised the plaster would look like this, but the plaster actually looks like that, so I want my money back. And that's one way that you can do this kind of dispute resolution. And then the ML model can determine whether or not the plaster looks closer to what they promised or whether or not it looks closer to something else. But besides that, you can also going back way back to some of this conversation, I mentioned that I am interested in a world where you can have privacy and not trust a bunch of people with these weird devices where you don't really know what's going on, but also prove authenticity. And the way to do this is that, assuming Apple and Google and the other phone manufacturers play ball, we can have a test of cameras on every smartphone that's being produced in the future. Then you can basically take an image of your face and say that, okay, I say, for example, batch an image on a driver's license. And you can do this using these ML models completely privately. Like, as, say, someone who is trying to log on to a service, I can literally reveal that I can only reveal that I am a citizen of The United States over the age of 21 and nothing else. And so this resolves privacy concerns for consumers and also even perhaps resolves some concerns about data gravity and privacy for service providers as well. So I'm hoping that this is logic is more widely adopted, especially with these attestive cameras.

Nathan Labenz: (1:06:47) So the way that that would work and this is kind of an inversion. Right? Because now we're saying we want use case we've primarily talked about is a commercial model provider. Know, we wanna be able to confirm in an individual case or kind of audit generally that they're doing what they said they were gonna do. Similarly, you know, in a in a context of, like, a dispute resolution, if it really matters to me, I wanna be sure that, like, the, you know, the evaluating model that I agreed to use is the 1 that is in fact being used. In those cases, like, I, as the consumer, don't have access to the weights. But now in this other scenario where I'm trying to prove my humanness, all that stuff is now happening on device. Right? So instead basically, my iPhone is running a process that is using, like, some ML model trained by, you know, whoever could be the government or Apple or whatever that that allows it to export a proof of something being human or something, you know, being attached to certain information, doing that in a way that doesn't reveal the inputs. So in this case, the model weights would maybe be known to all parties, but the inputs that I used are not. But I can still get to the point where it's like, okay. Yeah. So this computation was run, and therefore, you know, we can trust this individual user, you know, took a live photo of their face or whatever.

Daniel Kang: (1:08:10) Yeah. Exactly. That that, you the model weights here would be known to both, but the inputs will only be known to the person taking the picture.

Nathan Labenz: (1:08:18) I mean, a couple couple of little detailed questions and maybe just zooming out again to close. How does this change if we relax certain restrictions or kind of constraints on the problem? Like, if we were to say, instead of, you know, hypothesizing an OpenAI as the model provider, what if, you know, me and my, you know, home contractor agreed to some kind of open situation where the model that we're agreeing to use is kind of known to us? Like, we you know, there's a canonical version where we have the weights. In that case, you know, I was just kind of thinking thought experiment. Like, I guess, worst case, we could both run the model and kind of, you know, compare and contrast that way. In an open source model context, is there a need for this? I guess there would still be, you know, to to preserve my own inputs to a certain degree. But help me kind of understand that because, obviously, the open source, I wouldn't say it's quite catching up, but it's certainly achieving all sorts of new milestones where, you know, Llama 3 might be perfectly good for for various forms of dispute resolution and we'll probably have those weights.

Daniel Kang: (1:09:29) So with dispute resolution in particular, you can each individually just run the model. But the problem is that if you have a smart contract arbitrating the dispute mechanism, a smart contract needs to be able to run the model, so to speak. And as you pointed out, smart contracts running standard code is very expensive. So the zero-knowledge proof here would just be used to basically confirm the smart contract that the model ran in a particular way. Also, because these are large proofs to ensure privacy, you also might imagine if you are publishing something to a smart contract that's publicly on the blockchain, you might want to not, say, post pictures of the inside of your house publicly so that you can instead do that privately. And so there are other reasons why you might want to use these mechanisms regardless of whether or not you're using an open source model or not.

Nathan Labenz: (1:10:18) I mean, my crystal ball gets very foggy after the next 6 months. The fact that you are doing this work suggests, you know, a vision into the future that's at you know, and I don't wanna suggest that you are overconfident in your predictions, but you you definitely have a point of view as to kind of where the world is going and and what is needed. Could you kind of, you know, sketch out a little bit more how you think this impacts daily life? Like, are we interacting with these things all the time a few years from now? Does this pose a challenge to, you know, conventional legal systems if yeah. Because that's been the promise of blockchain, but obviously hasn't really materialized. But, you know, again, you put the smart in smart contract, things might actually get pretty interesting. I'd love to hear your just kind of license to speculate, granted. Tell me what you think we have in store for us.

Daniel Kang: (1:11:15) That's a great question. I wish I had a crystal ball then, you know, buy Bitcoin and tights, follow it. But maybe taking a step back. If you think about how cryptographic techniques are deployed today, they're basically deployed in ways that you and I don't really think about on a day to day basis. So we're recording this on a platform called Riverside, and there's this, like, walk that you see in your browser, which says that I have a secure connection to Riverside. And the underlying technology is called TLS, HTTPS. This is all run without you and I ever really thinking about this. I'm hoping that if the costs come down enough, it's basically the same thing. So for high stakes situations, like, you just won't think about it. It'll just be integrated everywhere. So if you take up an image of an X-ray or like a CAT scan or something, they'll just be like a assigned device there. It'll sign what comes off. You send it to the ML model. The ML model will produce a proof, so this is all auditable. You can always go back and just check what happened, which will make medical disputes not as much easier. But on a day to day basis, assuming everything is done properly, I won't even have to think about it. Similarly, for these smart contracts, the goal would be to make it so that disputes can be done automatically and ideally in a much cheaper way. I mean, if you look at the cost of liars, the high powered ones are quite expensive, so if there are any ways to reduce the cost and allow everyday people to use these technologies in this automatic way, I think that that's what this is moving towards. But in order for that vision to happen, the costs need to come down quite dramatically. We need to convince the hardware manufacturers to include these siding units, which can be quite expensive onto these onto your phones, say. But I think that as generative AI becomes stronger in terms of those capabilities, hopefully we will move towards this towards this vision.

Nathan Labenz: (1:13:06) What is the cost profile for these kind of on device mechanisms? And then I'm kind of also wondering, like, obviously, there's a lot of societal concern around deep fakes and, you know, information, disinformation, you know, scanning. We got, you know, spear phishing attacks on the rise. Does this seem like a policy intervention that might have legs? I mean, again, the cost would be pretty critical there, but people are looking, I think, very actively right now for what are things that we could kind of mandate that would give us some assurances about the future. And, you know, if if there was a single and and people are looking at choke points. Right? In the AI safety discourse, it's like, well, hey. Maybe we can sort of engineer certain things into the GPUs so that, you know, if they're ever part of, like, a super big cluster, they'll, like, phone home and tell people about it. And, you know, we'll see if that ever materializes. But here, there's something, you know, kind of similar where there is, like, a a serious choke point in a major you know, couple couple choke points. Right? There's only a few really big device manufacturers for, say, cell phones, you know, on which, like, the majority of the images are now captured. If the cost was, like, reasonable and it wasn't too onerous, it does seem like there could be an intervention there where you could imagine a new regulation that would say, hey. Apple, you know, Google, who Android, whatever, you guys, you know, you wanna sell these things, you gotta have this capability on there. I don't know if it would be the kind of thing that would be, like, annoying, you know, on a daily basis. The cost could be, like, hardware cost. It could be, you know, runtime cost for each image. It could be, you know, that, like, I can't take 10 pictures a second anymore, and I can only take, you know, how many pictures. So I would to understand those costs a little bit better and whether you think this is headed for a possibility of, like, a mandate of some of these technologies just to try to keep control on the information environment?

Daniel Kang: (1:15:10) That's a great question. I mean, so there's there's several costs here. So there's a hardware cost, there's also cost of, say, like, when you're taking photos live. So maybe to address the second 1 first, it's very easy to put bypass mechanism. You don't always have to go through the signing channel. So if you so you can basically engineer these things such that if I want to take pictures normally without this special thing going on, I can just do it. So then from a user perspective, no cost at all. From a hardware perspective, I can tell you actually that the Apple does come with what's called a secure enclave already. It's just not hooked up to the camera in a way that developers can access. And so basically, what needs to change is that they just need to hook up the secure enclave to the camera and allow some programmatic access. And the reason that they haven't done this is because, as you point out, there's no mandate to. And it does cost a bit extra in terms of engineering time and a little bit extra in terms of manufacturing costs. So this does sound like a perfect place for regulation to, or at the very least, nudge them in the right direction. And if any of your listeners or yourself happens to know people who want to think about this, I would love to chat with them, because I personally think that we should be moving to a world where we should have the option to authenticate any kind of media, whether it be audio, video, images, otherwise, to fight against things like generative AI. And this is something that these hardware manufacturers know how to do already. They just haven't because of the costs, which sounds like a great place for regulation.

Nathan Labenz: (1:16:41) In, like, dollar values, do you have a sense you know, I go pay 1000 dollars retail for an iPhone. They're obviously making some healthy profits on that. It seems like if this was a $1, you know, hard cost to Apple, it would be, you know, pretty easily absorbed in the system. Obviously, they're selling a lot of units, but still it's, like, not a big deal. If it was a 100, you know, you'd probably get real pushback. Do you have any idea what that incremental would be?

Daniel Kang: (1:17:10) So this is one of these things where people have very wildly differing opinions, and Apple is the 1 who knows their their internals the best. But if I had to speculate, I would say very marginal. They literally already have a secure enclave on every iPhone that's produced today. It it just needs to be wired in the correct way, and that's not it it's not that expensive to do that. Maybe in, the 1 to $10 additional cost is my guess.

Nathan Labenz: (1:17:34) Fascinating, really. Any other kind of visions for the future you want to tease or anything else that we didn't get to that you wanna make sure the audience understands?

Daniel Kang: (1:17:44) Maybe just a previous of upcoming stuff. So I mentioned already we're going to be publishing work in the next few months that will reduce the costs for certain kinds of models quite dramatically. We'll also reduce release some work also hopefully in the upcoming months on how to audit ML models in a very specific way. But that's all I had in mind.

Nathan Labenz: (1:18:04) Professor Daniel Kang, thank you for being part of the Cognitive Revolution.

Daniel Kang: (1:18:08) Thank you for having me. This was a this was a lot of fun.

Nathan Labenz: (1:18:11) It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.