Full Transcript

Transcript

Nathan Labenz: (0:00) Hello, and welcome to the Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my cohost, Eric Thornburg.

Hello, and welcome back to the Cognitive Revolution. Today, we have a short but fascinating conversation about investing venture capital into the AI wave and about the application of AI to hardware problems with Eric Vishria, general partner at Benchmark, and Sergiy Nesterenko, founder and CEO of Quilter. Regular listeners will recall that we recently cross posted an episode featuring Sergiy from Will Summerlin's autopilot feed, but rest assured that this episode covers almost entirely new territory, including what amount to my follow-up questions on Quilter's vision for superhuman circuit board design, the reinforcement learning approach that they're taking to get there, and what this could all enable for society more broadly. We also get Eric's point of view on what sorts of investments are likely to pay off in the AI era, starting with the provocative vision of circuit board design as an AI and done problem, not another copilot problem that attracted him to Quilter in the first place. We also consider the questions that Benchmark asks in an attempt to determine whether a new company is solving an enduring or a temporary problem, and we discussed the reasons that Benchmark has not invested in any foundation model companies to date. Eric describes these companies as investing huge amounts of capital to create some of the most remarkable, but also some of the most rapidly depreciating assets in venture capital history. As always, if you're finding value in the show, we'd ask that you take a moment to share it with friends. And at the same time, we invite your constructive feedback and your topic or guest suggestions either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Finally, a quick reminder that I am looking to connect with aspiring AI engineers and also AI advisers who are excited about helping main street businesses take advantage of AI technology. I've mentioned this twice and got good feedback each time, so please keep it coming. And before long, we might have a cognitive revolution jobs board on our hands. For now, I hope you enjoy this under the hood look at AI investment strategy and this glimpse into the future of AI powered hardware design with Eric Vishria of Benchmark and Sergiy Nesterenko of Quilter.

Eric Vishria, general partner at Benchmark, and Sergiy Nesterenko, founder and CEO at Quilter, welcome to the Cognitive Revolution.

Eric Vishria: (2:33) Thank you for having us.

Nathan Labenz: (2:35) Excited for this conversation. Regular listeners will, of course, know that we ran a cross post with Sergiy from Will Summerlin's new feed on all things AI in automation. And so for the deep dive on Quilter, that's the episode to check out to get up to speed and learn how the company is approaching the automation of integrated circuit board designs with a reinforcement learning approach, which is fascinating. Today, think kind of a little bit of a broader conversation on just kind of where this whole AI thing is going big picture and how you guys are thinking about both investing in companies that will hopefully stand the test of time and also trying to build a company that will be able to successfully ride the wave and not be crashed over by the wave, which is definitely a challenging dance for many founders right now. I guess, Eric, maybe you want to start off with just giving a little bit of background as to what it was that got you guys interested in Quilter specifically and how that fits into your broader philosophy and portfolio?

Eric Vishria: (3:37) Sure. Super, super excited to be here. So thank you again for having me and having us on the show. There's no shortage of AI startups out there. So we're each of us each of the benchmark firms are probably meeting 4 or 5 of them a week. And so, you're kind of perpetually meeting new ones. And a lot of them sound really interesting. You know, they're really interesting ideas. And I would actually even say the revenue traction on a lot of the companies is tremendous. Like, it's just unlike anything that we've seen, just in terms of the speed of revenue traction and everything else. But obviously, in the kind of timeline that we're investing as early stage investors, you're looking on a 5 or 10 year horizon, really, of the company maturing into a big business and wanting to be one of these exceptional kind of companies that get created in our industry a few times a decade.

And so a big part of what we're looking at is which of these companies is not just riding the wave and not just on a sugar high, as I would describe it, of either revenue or traction or developers or hype, investor dollars, but has an opportunity to build a really big business. And obviously, that's really hard to figure out at the early stages of the company, but that's what we're paid to do. So that's what we try to do.

And one of the frameworks that I use quite often is just, well, what's changing in the world that is gonna enable this company to be really large? Which has nothing to do with company, actually. It's just like what's happening in the world. And of course, there's AI, which is, that's a wave. But there's actually you need something more substantive and specific for a company to be able to be creative and be durable around that.

And I think in the case of Quilter, there were a couple of things that really caught my eye, which was, first off, obviously, electronics are permeating every aspect of our life, right? Like, things that were very simple not that long ago, like a light switch, now have electronics in them. They have circuit boards in them in ways that just weren't true before. And so you have a rapid expansion of the number of companies, the number of products that have electronics and have circuit boards in them. And that is a first very fundamental thing that's happening in the world and seems bad.

Then and I didn't really understand this until I met Sergiy, but the process of actually designing or laying out these circuit boards place and route, I guess, as it is called in the parlance is like an incredibly manual and time consuming and slow process. And one of the things I think we know just industry wide is the faster you're able to iterate on products, the better and better they get. And that kind of friction in the process seems bad in a world where electronics are permeating everything that we interact with. So that was interesting.

But then the real thing where it started to get where the story started to get really compelling to me was particularly, I don't know, 6 months ago or so, everybody was talking about copilots. Copilot this, copilot that. There's a copilot for lawyers. There's a copilot for doctors. There's a copilot for developers. And in each of those spaces, there's team companies that are emerged that are doing some form of copilots.

And Sergiy's kind of provocative statement at the time was like, this isn't a copilot problem. It's just not for copilots. Like, you don't want a human assisted guy to do printed circuit board layout. And I was like, oh, okay. That's different than anything that I've heard in the last few months. That's just a different view.

And so it's, well, why? And it's, well, if a copilot or an auto layout does 90% of the job, then by definition, the other 10% of these layouts in each which is they're all these thin lines that are connecting one component to another component. And they're all over the circuit board. As you look at it, you're like, wow, a human laid that out, and then that seems like really tough. And if I had to draw another line, like how am I gonna draw the line and not mess up everything else that's already laid out? Like, they're pretty dense.

And so it's like, this isn't a It's just not a human assist problem. It's a problem that should be AI and done. AI should just do it. Do it entirely. And in order to get there, we have to have a like a clean interface where we kind of get the design and architecture of it, and then we have a thing that can be spit out and go to manufacturing. But those interfaces exist in the industry. And so we sit in the middle and we take on that aspect of it and we just lay it out.

And we have to obviously start with simpler boards. We can't do a super, super complicated iPhone A14 to start. Like, we gotta start with something simpler and work our way up, but we can work our way up and just be AI and done.

So that was the first, like, really provocative like, just makes you think, oh, wait. There's a whole bunch of things that we just are, like, doing with humans and, like, copilot's vogue, but is copilot the right answer? Like, in a whole bunch of cases, like, copilot's not the right answer. It's just not. Like, we should just let AI do it, which is cool. That was a different perspective. So that was the first thing.

And then the second thing which really kinda got me thinking was, okay, everyone's talking about, like, large language models. And he's, yeah, we're not using any of that. That's not what we're using. Oh, okay. Well, that's different too.

And I'll come back around in a second, but that was his perspective was like, you know, that's not the right AI for this problem. The right AI is we design a game, we tell the game what the optimization problem is, and do we want to optimize for price? Do we want to optimize for performance? Do we want to have a really dense board that's expensive to build, but super power efficient? Or do we want to have a really cheap board that's like bigger and has more space on it? And if we actually give the user these controls, then the AI will be able to give them a range of solutions and they can kind of pick where they want to be on that solution set. And I was like, that's cool. That's a very provocative thing.

And so those were kind of the very specific things, like which is there's electronics permeating everything. There was a view that this is an AI and done problem, not a copilot problem, which was different. And there was a view of the right kind of AI to use for this particular problem, which isn't, again, the kind of thing that everyone's talking about. And as you kind of dug into it, it seemed like really cogent. And I know nothing about printed circuit boards and know very little about, obviously, specific technology. So you're kind of thinking about it and then you're trying to validate it and then you're talking to people and going after it.

But the biggest thing that just to kind of zoom out for a second, for any entrepreneur and anyone kind of thinking through these things, there's a Chris Dixon wrote this short post several years ago called the Idea Maze. And if you haven't read it, you should read it because it's a short but very powerful post. And I'll summarize it. What you want to do when you're talking to an entrepreneur is like you want someone who's been rolling around in this idea for a long time. Like they've been in the idea maze. And they've kind of gone down a path and they've hit a dead end, and then they've backed up and gone down a different path and hit a different dead end and then backed up and kind of tried to figure their way through the maze.

And as you're talking to them and asking them questions, what you keep getting is you keep getting, yeah, I thought about that, but here's why that doesn't work, so here's why I think this could work. And you're just like and what you realize is maybe they have a path through or maybe they don't, but they've been exploring and living and rolling around in this idea and turning it over and turning it this way and that way. And so they just there's so much depth on the problem. There's so much depth on the potential solution that you kind of are like, if there's gonna be someone who's gonna figure it out the way through the maze, like this person has a really good shot at it.

And obviously with Sergiy, his experience at SpaceX designing a bunch of boards, going through that, and then playing with and trying to figure out solutions and applications of AI to solve this problem. First realizing the problem, trying to pursue a whole bunch of different ways to solve the problem, having the realization that it really is an AI and done kind of problem, having the realization that of the right kind of AI techniques to use for it. Those were just evidence to me that he's lived in this maze for a long time. And so that was probably the macro thing that gets you excited. But it just ends up being very exciting when you find someone who has kind of approached it like that.

Nathan Labenz: (12:25) Cool. I appreciate the backstory. I kinda wanna maybe circle back in a minute to this disruptive approach. This seems like a pretty classic, almost textbook example of a disruptive solution in that it seems like it's coming in at kind of the low end of the market. It's serving people. If I recall from the other episode that we posted that it was striking to hear actually that SpaceX has an internal board team, but they couldn't serve you. And it's man, talk about a market that's gotta be very broadly underserved if an internal specialist team can't even serve the other team at SpaceX. Like, that's pretty wild.

So it seems like there's a potentially a very kind of textbook pattern here of starting at the bottom of this market, massively expanding the bottom of the market. I wonder how often that's something that you're seeing across the portfolio or specifically trying to do. But maybe before coming back to that, could can I try a little bit of this, like, idea maze stuff? I have a couple of ideas that I'm wondering kind of maybe why they didn't work or why they wouldn't work. Would you be game for a couple of possibly harebrained ideas that you can shoot down, Sergiy?

Sergiy Nesterenko: (13:33) Yeah. If I can, happy to.

Nathan Labenz: (13:35) Okay. These are potentially quite novice ideas. But I guess for starters, what's the data landscape in this space? Right? A typical deep learning approach is predicated on a lot of data. Is there any, like, open source dataset out there that somebody could go, like, tap into any significant scale of published boards that could be used in that way, or was sort of just lack of data a forcing function to make you go in another route?

Sergiy Nesterenko: (14:05) Yeah. That's a good question. And probably one of the most common questions that I get about, like, how Quilter is using AI for this problem. I think what most people tend to think is how does Copilot work? You take all of GitHub and all of Stack Overflow, feed it into an LLM, and it makes good predictions based on the average human behavior. And obviously, we don't do that.

So we don't do that for two reasons. Right? One that you mentioned is there actually just isn't that much data. Right? So if you look at all of GitHub, not that many open source boards. There's a few sources here and there, but not a lot. The best ones are locked away behind companies. Right? They're in Apple's repositories and Google's repositories and SpaceX's repositories.

But there's another reason that's even more compelling to me why that's the wrong approach, which is fundamentally, people are not good at designing boards. Just full stop. So if it's a process that takes for complicated board 2, 3, 4 months, you are going to make a whole bunch of, like, margin on margin decisions that make your resulting board, like, much bigger than it otherwise could have been. Use more layers than it could have, be more expensive than it could have.

And if you just use data to do supervised learning and try to predict boards, you're probably gonna get roughly that level of performance of a high level human designer, even if you could get all of that data. But with reinforcement learning, you have the opportunity to go significantly better than humans. Right?

So this is the most famous example I always come back to is DeepMind playing Go, right, the AlphaGo problem. They actually started first by training on human data and then creating agents that are based on human expert moves, and they got to a grand master level with that. But the best system today starts with no human data at all. It just learns how to play the game, and it determines whether it wins or loses. And then that is what gets you to far, far superhuman levels nobody can match.

Nathan Labenz: (16:47) So did you also start with, like, literally zero human data as input, or did you have some? And especially if you started with none, like, how do you get over the sparse reward problem? This seems to come up all the time. And I'm reminded also of the Eureka project. I used to say about AI, no eureka moments, meaning, like, at least from the generalist systems, you wouldn't see them doing, like, legitimately new stuff better than human. With the Eureka project, actually, I started to have to say precious few eureka moments because now there's at least, like, some examples that are starting to pop up.

In that case, I'm sure you're well aware they use GPT-4 to write the reward function for the robot hand as it was, like, learning to do all these tasks where in the beginning, its success is so fleeting or even nonexistent that it's really hard to even score what it's doing. So I guess I'm curious. How did you get over the sparse reward problem, especially given how little data you had to go on at the beginning?

Sergiy Nesterenko: (16:48) Yeah. Totally. So in general, the sparse reward problem kind of broadly stated is your good reward. So winning the game of creating a circuit board is so rare that as you randomly explore, you never find it. And that's an issue because if you never get a signal that you've won a game, how do you ever learn anything?

So the nice thing is that people have solved this problem. Right? Like, people in general have broken up the problem of circuit board design into many different steps and have basically come up with heuristics along the way. So what Quilter can do in terms of not running into the ultimate sparse reward problem of design an entire board, and at the end, only if you get a yes from all the physics simulators that this board is going to work do you get a 1, and otherwise you get a 0. Right?

What we do is just break it up into problems. Right? So we take the first part of the problem, which is placing the components. We can make sure that at that point, it's manufacturable, the components don't collide, things of that nature, and then compute some heuristics that basically indicate how likely we are to succeed at the next phase, which is the routing phase, and so on and so forth. Similar things for actually meeting all of the physics constraints. Right? Like, you can simplify the problem to an extent to compute basic, fast, dense rewards that correlate to your ultimate sparse reward. And that's what we have to do for now.

Now, will say that like long term, the dream for this is definitely like a single sparse reward. It's a single yes or no, this board will work or not. And when we're really uprooting the typical patterns that humans use and are trying to do much better than them, that's what we'll have to do. But for now, since we're not as good as humans at layout, especially on more complicated boards, we can still use the same heuristics that they would use to just at least automate similar to what they would have done.

Nathan Labenz: (18:27) Hey. We'll continue our interview in a moment after a word from our sponsors.

Ad break

Yeah. Interesting. It sounds like you're if I understand your comment correctly, the answer to this might be yes. But I was wondering if there's an analogy between the reinforcement learning processes that are used on the language models and these sort of evaluator systems, which I understand are like not models. Right? They're just sort of either simulators or like checkers that are deterministic.

But people worry a lot about in the context of language models, the idea that the human reward signal is not fully reliable. Right? Like, we're kind of inconsistent. We're sometimes mistaken. It's been observed in many language models that there's a certain like sycophancy tendency where it seems to try to tell you what you wanna hear versus the truth in some cases because maybe that's what got higher reward in the training process.

So it sounds like there is a similar problem here where the checkers are that they might be, like, purely physics simulators and could be like rock solid or I could imagine that they're on a foundation of sort of a bunch of heuristics which might in some way subtle ways kind of also lead the process a bit astray.

Eric Vishria: (19:37) One thing that I've this is an analogy that is imperfect in about 50 ways, but it's kind of supposed to like one of the things that I've been thinking about to try to articulate the Gen AI limitations, and particularly like the LLM and even stability models, like limitations versus some of these other techniques and what humans are actually good at is like I think humans are very good at extrapolation. So, like, coming up with novel things and developing and pushing creativity and if this, then that. Like, we can also do this and you can take it and extrapolate.

But because of the way these are trained and everything else, it feels and of course, who knows? But it feels like a lot of the Gen AI stuff is good at interpolation, which is within the bounds of things we already know, like, what are other points, like, in that space. And it kinda makes sense if you think about it, like, they're interpolating. They're figuring out, like, what the next word is or what the next image could look like based on the training set that is pushed in a bunch of dimensions and humongous and everything else, but it's still bound. It's bound by all the stuff that humans have prior created.

And so this is part of why I think that AI scare stuff is so misguided and not really because it's, hey, if they're interpolating, okay, that's looks like it seems fine. And it feels like humans will continue to be good at extrapolation, which is developing new novel techniques to do, like, various things. And that's a very important and valuable thing that we'll be able to continue to do. That was a very macro answer to your question. Curious what you guys think.

Nathan Labenz: (21:22) I have a ton of thoughts about that, but I think our subscribers have heard them in other contexts. I mean, it's kinda feel like all this stuff is converging, so that's why I was making the connection between how much basically, how much do we trust the reward signal in the context of reinforcement learning from language models? It's certainly helped a lot, but not that much, I'd say, is kind of the consensus answer.

I wonder what the situation is in the context of circuit boards. And then we could maybe also speculate about a similar question when it comes to self-play. There, it seems the sort of narrow problem, not just one domain, in general, narrow problems are better suited to self-play for now. But we are also starting to see some of these self-play techniques be applied to language models, and that's where I'm also like, I don't know if it's gonna stay in the bounds of what humans have given it for all that much longer.

But let's take it piece by piece. Sergiy, let's start with the how much do you trust the reward signal in the context of your problem?

Sergiy Nesterenko: (22:31) Yes. One of the nice things about working on a hard physics problem, like, humans are not in the loop. Like, humans are actually bad at looking at a board and judging if it's going to work or not. That's where, like, 80% of boards that are built are faulty in some way, and it's because humans are just not good at that task fundamentally.

But all of the core physics is computable. Right? Like, we care about laws of the the Maxwell equations and laws of thermodynamics. And we've had convergent techniques to solve those for over 100 years. So in our case, like, if you actually use the Oracle, like, actually numerically solving the Maxwell equations for every set of possible considerations and problems, and you're kinda being careful to make sure that you're convergent and approximate everything correctly, it's completely trustworthy. Like, much more trustworthy than the human result by far.

Only problem with that is speed. Right? If it takes 20 minutes to compute a single physics solution and you need to do millions of those as you fine tune on your model, that's problematic. So what you could do is you can make approximate models that are just conservative. And so at that point, maybe you're not doing the best possible arrangement that physics could allow, but you're still doing one that works, still doing one that competes or beats humans, and it's still definitely going to work because you've been careful about the physics approximations you've used.

So I think that's one of the luxuries that we have is that we don't have to negotiate with humans. Right? We just have to make sure that it's manufacturable and that physics tells us the answer. And physics is unambiguous about that.

Nathan Labenz: (23:54) Gotcha. And these were tools that existed already in the industry that you're able to build on top of. When the system is going about the process of designing, how iterative and sort of tree search-y is that process?

Because I think folks again will know at least the basics of AlphaGo. Right? These are, like, hyperparameters ultimately that you can sort of turn up or down at runtime. Right? But, like, how deep are you going to allow a system like AlphaGo to search the space of possible moves is a huge factor in how well it's gonna do. If you just make it do a single raw guess, it's not gonna do super well. If you allow it to map out a bunch of possibilities and then it gets scores on all those possibilities, then it can sometimes land on superhuman results.

So give us a sense for kinda what that search iteration and scoring loop looks like in your context.

Sergiy Nesterenko: (24:50) Yeah. I think, actually, this is somewhere where a user has meaningful choice. So for a lot of designs, like, you just want it fast. Right? Like you're maybe primarily focused on making sure that your schematic is okay. You don't care if the board is rather sparse, big, not very dense, maybe it's a little more expensive to manufacture, so on and so forth. And in that case, like you don't really want a whole lot of search. You just want the first answer that's going to faithfully implement the schematic on the board. And so in that case, the ideal thing is to return an answer within, you know, a few minutes to an hour.

On the other hand, suppose you're looking at a board that's gonna have a 100 million units produced, like an iPhone motherboard or something like that. And if you save a cent or a dollar in every board, like, that's a lot of value. In that case, you might let it search for a month and explore in all sorts of directions because that's how long it would have taken a human to do it anyway for a single design. Never mind for the billions and billions you could explore with this kind of system.

So this isn't like a lever that we have in the tool today. We kind of just we basically treat overnight as the constraint for us right now. Right? So the idea is that at the end of the day, you finish your schematic, you upload it, and either immediately or sometime within the next 12 hours, you get a result in that morning. The next morning, you can look at it and see if it meets your standards. But in the future, I see that being a lever, and it's for you to decide what is more important to you.

Nathan Labenz: (26:09) Yeah. That makes sense. What are the size of the models? What does the compute look like, and how much of it is on the inference side versus on the scoring side in the physics simulations?

Sergiy Nesterenko: (26:21) Yeah. Those are all really good questions. So, obviously, we're not dealing with the kind of compute that LLMs deal with. Like, we're not using 10,000 GPUs to train across 1 trillion tokens and whatever crazy numbers are being used nowadays. This is a relatively focused problem, and so we can deal with much smaller models. We can deal with much faster convergence times, much less compute effectively.

So right now, the vast majority of the compute is going into kind of actually playing the game, and so a lot of that is still actually CPU cores. And then some of it is GPU cores, depends on which part of the problem. And then the kind of other part of the majority is going into training. And we actually train during the production runs. So we have fast enough environments and fast enough evaluation that as somebody uploads a board, we're not just doing inference. We're actually training on that board as it was uploaded.

Now the cost of physics is going to increase for us for sure. One of the ways that we are expanding in the market is by enumerating all of the different types of physics considerations you have to look at and basically chipping them off one by one. Right? So instead of saying, hey. We're gonna attempt all kinds of boards from all types of physics and do only the easy small ones, we're saying, okay. For now, we're doing low speed boards that have up to 4 amps of current or something like that. And, okay, the physics to compute that is straightforward.

The next thing we're working on is high speed digital, and then we'll step into approximate, maybe bounding methods of computing whether or not those high speed digital signals are gonna be okay or not. That won't be too expensive. But then we'll do one or two validation runs at the end with a full wave model. And that's probably if I had to guess right now, it's probably gonna cost us a few GPU days per board, something of that nature.

Nathan Labenz: (28:00) Interesting. Okay. So it sounds like ultimately more compute on the validation side than on the generation side just because of the intensity of simulating. Is it simulating physics, or is it just solving is it like a closed form solution of a ton of crazy differential equations, or is it a sort of more Wolfram style, like, you gotta actually play this out in simulation, no shortcuts kind of a thing?

Sergiy Nesterenko: (28:28) Yeah. The simulations, that's another thing that we're exploring heavily. There's a lot of different ways to solve differential equations. Right? So the most brute force simple way is, at least for electromagnetics, this method called finite difference time domain, where you literally just grid the entire world in 3D, and you basically just apply the differential forms of the Maxwell equations in sequence. Right? So you take almost like a curl of your local pixels of electric field and do that for a magnetic field, do that for electric field, and you have to do that for 100 million cells for 100,000 steps, something like that. So it's just a lot of raw compute.

But there's also much more clever methods. Right? So FEM is a different way of approaching this problem within electromagnetics in particular. There's things like method of moments that only look at, like, the surfaces of your different electrical systems. And then there are approximations. So in the approximation of a low speed, low frequency, you can actually factor out time out of the differential equations. And so you can do this thing called the quasi static approximation where you don't run time at all, and you're only looking at capacitance and mutual inductance of the systems. And for certain frequencies, that's a perfectly sufficient approximation, so on and so on and so forth.

Nathan Labenz: (29:35) Yeah. Very interesting. The sort of maybe that reality maybe blunts the value of this next idea that I had, but I was thinking if you did have a lot of data, and I wonder presumably there are some I don't know the structure of this industry, but just using, like, chips as a reference point, obviously, there's a few manufacturers that take in a lot of designs and output the actual devices.

I would assume there's probably some big players in the circuit board space as well who are, like, getting lots of designs. And one might imagine the sort of big data approach, something like a diffusion model seems like it could be an interesting fit for this where you would kind of run a bunch of loops through whatever so the models can vary. But say you have a transformer, like the latest stable diffusion version. They run a bunch of passes through this transformer. At each step, they're denoising from raw visual noise to the image. You could imagine sort of a similar approach to and this is also happening for proteins now too. It's crazy.

So it seems like it could also perhaps apply to a circuit board design. And first, you're kind of doing it would sort of even follow the path that you described where it's first you kinda lay out the big things, and then gradually, you're getting more and more into the low level details of the design.

I guess questions there would be like, do you think that would work? And if so, is that something that worries you from a competitive standpoint at all? And then maybe though, it just wouldn't be that much of an advantage because if all the compute is on the simulation side anyway, then maybe it doesn't really matter.

Sergiy Nesterenko: (31:20) Yeah. So the big issue with that is the quality of your data. Right? So one of the just the facts of life in this industry is that 3 times out of 4 that you submit a board to a manufacturer, it looks right. It's manufacturable. The manufacturer can follow all the tolerances and all that stuff, but the physics just doesn't work. So even if you collected all of that data, you still have the problem of going through and identifying which of these is an actual working board and which of these have mistakes. Right? Because it's a junior engineer or somebody who didn't see some sort of issue or something like that. Even senior engineers make mistakes on the electromagnetic supports all the time. I mean, you just you fundamentally have thousands of components, tens of thousands of traces to look at, all of which impact the other. And so it's just very difficult for a human to keep all that in mind.

So if you're gonna clean the data, you still have to run the simulations to make sure that whichever candidates you're training are actually good. The other problem is that the information that the manufacturers are getting is not sufficient to do this. The manufacturers basically get you can imagine this like a photo. Like, if you're developing film, you get a mask, a set of masks that shows you how to etch copper on all the different layers of the board and then which components to glue down. But that doesn't actually tell you what signals are happening throughout the board, and you need to know that to evaluate the physics.

So, I mean, we could generate our own data by just self-play or, like, every time we find a good candidate, save it in the database, then train the diffusion model to just recreate those. I think that's valid. But the point is that you still have to kind of come up with those data points that you verified from physics first principles are actually good designs.

Nathan Labenz: (32:58) Yeah. That's interesting. I feel like there's some way in which the noise sort of cancels out. No doubt that in and I'm just kind of porting my intuitions from other domains of AI here, but certainly no doubt that people do spend a lot of time curating data and going for quality. But also, it does seem that the models are pretty tolerant to at least some amount of kind of wrong stuff in the data. And I guess it kind of regularizes out in the training process, one hopes. And in practice, it does seem to work.

I don't know if you think that it's just, like, fundamentally not gonna work in this case, but it seems like more your motivation, if I understand correctly, is, like, it's more about you wanna get to superhuman, and you think that the reinforcement learning is obviously, like, the proven path to get there? And that argument definitely makes a lot of sense to me.

What do you think is the timeline to superhuman circuit board designs? What would keep you from going faster toward that superhuman board design future?

Sergiy Nesterenko: (33:56) Yeah. I mean, in my perspective, the bottleneck is talent. Right? Finding really great people to work on this kind of problem is the hard part. It's you have so many different aspects of this that need to be really amazing. Right? You need to have amazing people who are expert at neural nets, expert at cutting edge reinforcement learning methods. You can't just grab the latest thing and apply it and hope for the best. There's a lot more nuance to this.

But also on the C++ side, the CUDA side, the physics side, all of those things have to come together. The timeline, I can't predict it exactly. Maybe for small boards, we're a few years away. Maybe for something like a motherboard, we're 5 years away, but I'm guessing.

Nathan Labenz: (34:33) Yeah. My crystal ball also gets very foggy more than a couple months out.

Eric Vishria: (34:38) Yes. It should.

Nathan Labenz: (34:39) In terms of the kind of big problems that you need to solve that you're, like, maybe not sure how you're gonna solve or that you need the talent to come join the company to be able to get over certain humps. I often feel like when I talk to people, especially those in research and you may say, well, that's the difference is that it's research versus, like, actual engineering. But I often feel like I get the sense that like a lot of things are working, like a very high percentage of things are sort of working and that a lot of times like multiple different approaches probably could have worked. It seems like just in general, we've kind of hit on a couple of architectures that are really working, but it seems like there's a lot more where that came from and presumably we'll be discovering more and more all the time.

Is there something that you're like legitimately not sure if it's gonna work or really have no idea how you will make it work? Or does it feel like the kind of thing where, of course, there's going to be like work and optimization and making it run faster and all that kind of stuff, but basically it feels like you're going to climb. Is there like a sheer face of a mountain that you have to scale vertically or is it kind of graduated stairs that you're pretty confident you can climb one by one?

Sergiy Nesterenko: (35:53) I'm confident in the latter. Like, this is 100% a solvable problem. Given enough time, I'm confident we'll solve it. There's a lot of the nice thing about this problem is there's a lot of steps you can take. Right? Like, with something like self driving, you kinda have a do or die. Right? Either you're confident you're not gonna crash or you're not. And, of course, there's still gradations, but the stakes are really high.

With us, we have a lot of checkpoints along the way. Right? We have checkpoints in terms of complexity and size of board. We have checkpoints in terms of the physics that we can solve. We have checkpoints in terms of the ambition to be significantly better than humans that can come over time. And with that kind of those kinds of stairs ahead of you, you can treat each one as, okay. Like, now you need to make this 20% better, 20% better, 20% better, 20% better, and let it compound over time. I don't see anything fundamentally about this problem that is unsolvable in any way. It's gonna be hard. Don't get me wrong. It's gonna take a lot of people. It's gonna take a lot of effort, but there's nothing about this that seems unsolvable in any way.

Nathan Labenz: (36:51) Hey. We'll continue our interview in a moment after a word from our sponsors.

Ad break

When you hit that mature phase where we now have superhuman circuit board design, what does that look like to somebody like me? Can I just show up and say, hey? I'm making a talking stuffed animal. And like, how ignorant can I be and still get a board out? Because it's like today to even specify what you want is sort of an expert problem.

What do you see the sort of independent entrepreneur, tinkerer kind of person who, like, has a product in mind, knows literally nothing, do they talk to a language model and work their way towards specifications and then put specifications into your system? Or what's that sort of future state round trip look like?

Sergiy Nesterenko: (37:39) Yeah. Sure. So with us, like, we're focused explicitly on layout, which is kind of one of two problems. Right? To give you an analogy to hang on to, think of creating a schematic for a circuit board as writing code. Right? Like, a schematic literally looks like a block diagram with inputs, logic in the middle, and outputs, and it's entirely kind of logical and abstract. You're just communicating to other engineers what the inputs, outputs, and functionality of the board will be.

Layout is like compiling your code. Right? Like, how do you actually make something physical that takes that schematic or takes that code and, like, actually makes the atoms of the world do that? Right? So we're very specifically targeting the layout portion because we think a compiler should exist for electronics, and it just doesn't.

The nice thing about that is that a lot of the benefits that we now have because compilers exist in software, I think are going to happen and are gonna answer your question in electronics. Right? So compilers eventually led to higher level languages, eventually led to things like Python, eventually led to large language models that allow you to write the Python and automate the whole thing. We will eventually move up that stack and look at schematic and how to make that piece easier so you don't have to learn how to write C++ that you could learn how to write Python or maybe even just deal with block diagrams and make a circuit board.

Nathan Labenz: (38:54) Yeah. That's really interesting. Reminds me a lot of we just did an episode on this tiny GPU project where a young guy set out to design his own GPU from scratch in 2 weeks and ended up taking him 4, which was pretty remarkable. And a big part of why that's possible and it's still quite an accomplishment in my view, but big part of why that's possible is that the sort of equivalent of the layout compiler does exist. And so he was able to kind of get to that stage and be able to feed it into an existing system that could do that sort of gnarly work for him.

But, yeah, that's helpful. Just zooming back out to the Benchmark portfolio more broadly, what would you say here are kind of like the patterns that are common across the portfolio? What investments have you guys made that sort of have a very different pattern in terms of what part of the market they're going after first that are maybe in or not in this classically disruptive mode? And I guess broadly, how do I make money in AI?

Eric Vishria: (39:53) Yeah. That was the right question. I'm very bullish because I think the overall, like, we wanna be in areas where there's lots of disruption and lots of things changing. That creates the kind of primordial soup for there to be, like, new big things created. So it's really valuable in that way, and we've seen that. And there's always a lot of crap gets created and a bunch of stuff that doesn't work. But there are a bunch of good things that get created in the process, it's all about finding them.

Really loosely, I think there's you kind of have in AI right now, it feels to me like there's 3 big categories of companies. There's foundational model companies, right, that are like they're building foundational models or they have some techniques around it. They have proprietary data. Maybe they have other things that they're using and they're trying to do something really special there.

Then there's a set of infrastructure-y companies in the next category. I'm on the board of Cerebras, which we invested in in 2016. That's an AI chip and systems company that's focused on training. But you have Groq. I'm also on the board of Fireworks, which is an inference provider. And there's a bunch of things we're also investors. So there's just quite a few in that category too, which are their infrastructure, which is enabling some of the other stuff that's happening in the ecosystem. That's been a pretty fruitful area. I think those companies have done well. They've gotten revenue quickly and so forth.

And then the third category are companies like Sergiy's, like Quilter, which are vertical applications of AI. So they're applying AI to try to do something, right? And we see those for lawyers, we see them for doctors, we see them for accountants. We obviously print circuit board designs. So there's just like a bunch of those things too.

And each of those 3 categories, which is it's very abstract and loose and they're all kind of different, they have a set of really big opportunities and really big problems, I would say.

The foundational models are incredibly expensive to develop. They are extremely quickly depreciating. I said this before, I think they're the fastest depreciating asset in the history of humankind, which is like, you build one, you spend $150 million on it, and 6 months later, someone can build the same thing for 5 million. Like, that's not historically a good way to make money with venture capital. Like, that money tends to get incinerated and then someone has something alternative. And we'll see. I could be totally wrong on that. Maybe someone will build something that defeats it. But the general purpose models have proven to be really expensive and depreciate very quickly what is cutting edge.

The infrastructure companies have done pretty well in a bunch of ways. Like, they have real business opportunity. They're enabling a bunch of things. The question always exists with them, which is do those problems exist for very long? Like, to some degree, a lot of AI development is where software development was like circa, I don't know, 2002 in terms of like the tool sets available and the abstractions that people are working with. And so some of the things that people are solving, I'm just not sure that they're gonna be around for a long time. And so that's a potential real challenge there, but there's a lot

Nathan Labenz: (43:12) What do you even mean there? Are you talking like LLM observability type?

Eric Vishria: (43:17) Yeah. Like, that's a great example. That's a great example, which is just like, is the LLM observability thing a thing? Like, it's a thing today. I don't know if it's a thing long term. Like, maybe not. We have to kinda think through it. But that's that could be a potentially challenging area. That'd be one that we think about a lot.

And then the vertical stuff has proven to be the fastest revenue traction. Some of the revenue scale of the vertical companies has been insane. But the kind of common criticisms of, are there any moats or barriers to entry? If you can build a company very quickly in 3 months or 6 months and get something out a weekend hackathon for that matter. Yeah. Weekend hackathon. Then that also means that 10 other competitors are going to do that. And, you know, do you end up with something durable there or not?

And I think we just don't know. And in a bunch of cases, you have to have a theory of it. Like, in the case of looking at Quilter or our investment in Sierra or some of these others, like, I would say, we have a view, a belief that there's something durable there that will be built and compound over time, even though they got quick traction.

And so I think that's just something that we have to look at, and there's lot of value. This is what's cool about what's happening right now is there's like lots of traction. There's lots of really interesting ideas in all of these categories, and obviously amazing people working on them, but we gotta figure it out. And so that's kinda how I've been thinking about it.

I'd also say there's this other thing that's really interesting to me where a lot of the AI work has happened with researchers. Like you hear it all the time. This is a researcher, they're PhDs. They're doing that. Who's And telling me this? I forget. One entrepreneur was telling me this, like, so the researchers are writing something. It works. It kinda works in a research context, which is a proof of concept or experiment. And then they take that code and they try to scale it in production and it's, holy shit. That's not good. That doesn't work. It's not written for that or whatever.

And this entrepreneur said she's I have a rule when it comes to code written by researchers. And I think she said she was just like, it's just rm star. There's this bridge and divide, which is quite from between research and proving something out conceptually, and actually like turning that into shipping product that can scale and work and be iterated on. And that's a divide that just didn't exist in software development, at least in my adult lifetime. Maybe it existed in software development, like back in databases in the seventies and eighties, but like that, I don't know enough if it did or not. But it's been like it's so that's when I say that, like, the stack maturity, like, the kind of stuff that people have to do to work with PyTorch today or work with CUDA today or something like that. That's just not something that software developers have had to deal with for at least 20 years.

Nathan Labenz: (46:21) Yeah. That reminds me of comments I heard Demis Hassabis make about why they have taken this step now of merging DeepMind into Google proper after so many years of kind of holding it out as its own thing. He basically just said, we're now at the point where we're not done with research, but certainly a lot of the research has been done. And now it's like, becoming the bridge to engineering that is a huge challenge, and that's where they feel like it's the time is more of everything.

Eric Vishria: (46:48) We like we do want to continue to do novel research and push the limits of it. But as we try to bring this stuff into real applications to benefit humanity, we have to engineer it. And that's its own thing. It's not so simple, it turns out.

Nathan Labenz: (47:05) I know we're just about out of time with you guys today. Do you have any kind of call for startups? Anything that you wish somebody had brought you that you haven't seen or haven't been able to find yet?

Eric Vishria: (47:15) I have this thing that if a venture capitalist has the idea, with very few exceptions, like maybe Mike Speiser or someone else like that can pull it off. But if the venture capitalist has an idea, that's a bad situation, because this is not what we're to the idea maze point where we started. That is not our job, nor is the thing that we are generally good at.

Look, I would say I continue to be really excited. I think we have a ton of companies across the board in at least in terms of infrastructure companies and the vertical applications that are super interesting. Would love to meet amazing people who have been working on ideas or thinking about something for a long period of time and are like obsessed with it. And that's a very broad remit, but our job is to be involved in the best companies as early as possible. And so that's what we want to do.

Nathan Labenz: (48:17) Yeah. I hear that. With as fast as things are moving today, there's really no substitute for obsession.

Eric Vishria: (48:24) Yeah. Total.

Nathan Labenz: (48:25) Alright. Great. Well, this has been a lot of fun, guys. Sergiy Nesterenko, founder and CEO of Quilter, Eric Vishria, general partner at Benchmark. Thank you both for being part of the Cognitive Revolution.

Eric Vishria: (48:36) Thank you so much.

Sergiy Nesterenko: (48:37) Thank you, Nathan.

Nathan Labenz: (48:38) It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.