Full Transcript

Transcript

S0 (0:00) Forecasting the future is quite hard. We are having to make decisions about these things. The best guesses that we can get are valuable, and so it's important to hear what AI researchers think about this. Amidst quite a lot of uncertainty here, The chance of things just staying pretty similar and nothing coming of this to drastically change people's lives seems quite low. That seems like a pretty important thing to be keeping an eye on and trying to have accurate opinions about. What the public thinks about these things will affect what policies happen, and they think that, you know, could change the rest of the future forever.

S1 (0:36) Hello, and welcome back to turpentine AI. Before today's show, another quick reminder to take a second to sign up for our new cognitive revolution feed. Our latest episode on that feed is an outstanding conversation with the great Linus Lee of Notion AI. So find that and be sure not to miss any of the new episodes that will be exclusive to that feed by visiting our website, cognitiverevolution.ai to subscribe. Today, I'm excited to share my conversation with Katja Grace, founder of AI Impacts, who recently published what I believe to be the most comprehensive survey of elite machine learning researchers' expectations that has ever been conducted. As we all try to make sense of this AI moment and understand where things are going, Katja's survey provides an incomparable resource. With over 2,700 respondents, all of whom have published in 1 of the top 6 AI conferences in just the last couple of years, It offers a nuanced and detailed picture of what the people closest to the technology really believe. And the results are truly striking. Arguably, biggest takeaway is that there is no consensus expectation within the field. Most individuals express high uncertainty on the most important high level questions, and only relatively small minorities confidently predict a very positive or very negative future. Researchers definitely do not dismiss existential risk with median estimates of a 5 to 10 percent chance that AI could lead to human extinction. And meanwhile, timelines for key capabilities have shortened since the survey was last conducted just a year ago. There is strong agreement that AI safety should be prioritized more than it is today, perhaps in part because no major mechanistic interpretability breakthroughs are expected in the near term. Overall, this conversation offers a uniquely rigorous field wide take on critical questions that are all too often poorly asked and sloppily debated. After first digging into the methodology to better understand how Katja and team went about defining their terms, framing their questions, and summarizing the responses into aggregate views, we go on to consider implications for policymakers, the public, and the field itself. While there's, of course, nothing to say that ML researchers will ultimately be correct about AI outcomes, I would call this a must listen for anyone who wants to ground their own personal AI world models in the considered opinions of the people who are collectively inventing the technology. As always, if you find value in the show, please share it with others who you think would appreciate it. I genuinely believe that both the doomers and the eax should update their p dooms toward these community estimates. A 5 to 10 percent chance of extinction is obviously a very critical issue, but the shape of the other 90% of outcomes is also well worth worrying about. Of course, I'm always eager to hear your feedback, so please feel free to reach out via your favorite social media platform. And now please enjoy this exploration of the expectations and the substantial uncertainties of the AI research community with Katja Grace of AI Impacts. Katja Grace, founder of AI Impacts. Welcome to the cognitive revolution.

S0 (3:55) Thank you. Great to be here.

S2 (3:56) I'm excited for this conversation. You have just recently completed 1 of the most comprehensive surveys of the worldviews and expectations of elite machine learning researchers. And as we try to grapple with this AI moment and try to make sense of what's happening and where things are going, I think it's a really incomparable resource. So I'm excited to just spend the full hour kind of digging into that and understanding both how you went about collecting this opinion and also what the aggregate opinions are. For starters, you wanna just introduce maybe yourself and AI Impacts a little bit and and give a little bit of the inspiration or motivation behind the survey?

S0 (4:38) I am broadly interested in trying to make the world better. And for the last 10 years or so, I've been involved in trying to figure out what's up with the future of AI, since it seems like potentially a pretty important thing to try and cause to go well. I've been variously skeptical about whether it's likely to cause human extinction or not, but, yeah, interested in figuring out these kind of things. So AI Impacts is an effort to try and answer questions like this and just various high level questions about what will happen with the future of AI that are relevant to decision making for lots of people. Is AI likely to cause human extinction? When will the big AI thing be? What kind of thing will it be? Are there other kinds of changes to society that we should expect and do something about? AI Impacts is often answering sub questions or sub sub questions that are relatively in the weeds as input to those things. For instance, case studies about other technologies and whether they've ever been slowed down for ethical or risk reasons and what that looks like for a recent example. So we do a lot of stuff that's not directly about AI, but is intended to be part of a bigger network of questions that inform other questions.

S2 (5:57) Cool. For the purpose of a survey like this, is there any reference point that you can look to where folks have found it necessary even to survey experts to understand the direction of their field? I would think maybe something like biotech might have something like this, or you could imagine I just recently read a story about the original nuclear test at Los Alamos and how they had a little betting pool on it, but there was no, like, big survey. So are there any kind of other touchstones that you look to?

S0 (6:29) I think the the thing that we most look to is just, like, other surveys in AI over the years where prior to the 3 that I've been involved with, there were older, smaller ones that often have a different definition of human level ish AI each time or something, so it's a bit hard to compare the results. But as I previously looked for all the things like that we could find and go back to, like, I think it was 1 of the 70s. Yeah, interestingly compared to over the years, I guess we're not looking to them that much methodologically, because I think you're maybe less careful and smaller. In terms of things in other fields, I haven't looked in detail at them. My understanding is that there's some sort of thing in climate perhaps.

S2 (7:13) Yeah. That's a definitely good reference. There's certainly a lot of guessing as to the future state of the climate. So, yeah, that makes a lot of sense. Okay. So the way I plan to proceed here is just, first of all, talking about who are you surveying? Like, who are these people? Then what was the survey that they took? A little bit of on just the structure, the experience, some of the key definitions and framings that you used. And then I think the bulk of the discussion will be, like, what are the takeaways? What do people actually believe in the field? But for starters, who were these people? How many are there? And how did you get them to do it?

S0 (7:46) We tried to write to everyone who published in 6 top venues, so 5 conferences in 1 journal in 2022. So as in in 2023, we reached the people who had published in 2022, and we offered them $50 each. We had a little trial at the start where we offered nothing, but some of them and $50 each to another small group. I mean, it looked like the $50 each was doing a lot better, so then we offered it to everyone. It was a decent task to dig up their addresses. We got their names from all of these papers, and many of the papers have emails for at least 1 of the authors, maybe several of the authors. We dug up the other ones from other papers of the internet somehow, and I think we found them for a large fraction, like more than 90%. I have to get the exact number. So I think this is, like, in the range of 20,000 ish people, and we got about, like, 2,700 to respond.

S2 (8:39) These are people that published in 6 top conferences. Is there a natural reason that you chose the 6 conferences? Like, why not 4 or 10?

S0 (8:51) So previously, the last 2 surveys we've done, we just did in Europe's and ICML, and we wanted to expand it partly to wider range of topics. So I think like AI people who are not necessarily all doing machine learning and also just to like cover more of the top people. So I think it ended up being 6, because as far as we could tell, talking to people around, that was a sort of natural set of them to do. But, yeah, I I don't have a clear explicit description of what makes them the ones.

S2 (9:25) Is there a time horizon also on how recently they published in these conferences?

S0 (9:31) It was everyone in 2022. Like, everyone who published in 2022. Often, people published in multiple of them.

S2 (9:37) Quite recent. Yeah. Yeah. Okay. Cool. So we got 6 conferences, universe of 20 ish thousand people, something like a 15% of respondent rate driven in part by a $50 gift card, 2,700 plus total respondents. First of all, interesting and there's, of course, a whole paper here. You guys have a lot of visualizations of data in the paper, so definitely encourage folks to go check out the the graphs if they wanna do a real deep dive, although we'll cover the bulk, I I hope, of the headline results. But it's kind of a branching structure where, like, not everybody's getting every question, but you have some anchor questions that everybody's getting. And then others, there's kind of a bit of a not choose your art adventure, I guess, but kind of a randomly chosen adventure for them. And it takes 15 minutes, which it was kinda surprisingly fast. I would have expected to go slower.

S0 (10:30) Yeah. I I guess we would actually check for for this 1 how long it took. That was, like, an estimate estimated based on, like, the the previous 1.

S2 (10:39) Advertised as 15 minutes. Was there any free response, or it's all structured? Like, you must either end like, select a a thing or, like, enter a number. Right? There weren't, like, any paragraph style responses that I could detect?

S0 (10:52) There were some that weren't paragraph length. There was just, like, some some brief ones. For instance, there was 1 that was, what's an occupation that you think will be automated, fully automatable, pretty late, something like that. So that's a couple of words. But also, for I think all of the sets of questions, you could randomly I think maybe 10 of people after each 1 got there are 2 different questions you could get about just like, what were you thinking when you answered that? Like, how did you interpret this question? Which are not part of the main results, but for us to be able to go back and if we're confused about a thing, check if everyone's misunderstanding it or something like that, which we have not gotten into yet, but we might do more of. Oh, yeah, actually, sorry. Yeah, there are various open ended ones. There's another 1 that was like, what confusions do you think people have about AI risks, something like that.

S2 (11:42) Okay. Maybe we can talk about some of the open ended stuff later. And I I'd be interested to hear how you may think about evolving the the survey in the future, particularly as you can potentially bring language model analysis to open end responses. I I'd expect surveys in general are gonna be an area that will change quite a bit. Certainly, I see that in even just very sort of application centric customer feedback experiences. I'm like, hey. Let's just let them talk. We don't need to structure everything in a grid or rubric quite as much anymore. We can if we're looking for insights to some degree, you can just let people sound off. But for these purposes, you're really trying to get to quantitative estimates. And as I went through the paper, there were 2 super high level framing structures that stood out to me as, first of all, just very thoughtfully done and and worth understanding for our audience as well. 1 is that you have 2 different definitions, essentially, I would say of AGI. Right? Like, there there's 2 different ways of thinking about, like, powerful AI that we definitely don't have now, but we could have in the future. And then there's also 2 different ways of asking people when they expect certain things to happen. Maybe you can unpack, first of all, the 2 different definitions of AGI if that is a reasonable way to think about it.

S0 (13:00) Yeah. I think so. I guess I think of those fairly closely related, but, yeah, different people think of that differently. So there's high level machine intelligence, HLMI, which is like when unaided machines so it doesn't even have to be 1 machine, just like any collection of AI can accomplish every task better and more cheaply than human workers. That is, like, for for any particular task, there there are machines that can do it. It doesn't even have to be the same collection of machines that does stuff. And we're ignoring aspects of the task to which being a human is intrinsically advantageous, e. G, being accepted as a jury member. We're asking them to think about the feasibility of it, not whether it's adopted. So this is roughly like, when can AI do all tasks better than humans? Then the other definition is full automation of labor, which is sort of similar, except that it's asking about occupations instead of tasks. So occupation becomes fully automatable when unaided machines can accomplish it better and more cheaply than human workers, Ignore aspects of the occupations, which being a human is intrinsically advantageous is the same. And again, asking them to think about feasibility, not adoption. So I think there's some chance that keep all of these things in mind and tend to think of the automation of occupations as actually about adoption and the high level AI 1 about it being feasible. So when all occupations are fully automatable, then we have full automation of labor. So I would think of an occupation as probably like a big complex task, or if not, composed of many smaller tasks. So I would think that if every task is automatable, then that implies that occupations are automatable as a subset of tasks. But in fact, the answers that we see put the date for HLMI much earlier than the date for full automation of labor. So I think there's a question there about what people are thinking. And to be clear, no 1 is answering both of these questions. People are randomized to receive 1 or the other. So you can't say to a particular person, like, why are you so inconsistent? But given that there's such a big gap, presumably, any given person is likely to be inconsistent if you ask them the 2 questions because it's not just random or something. There's 1 other small difference between them, which we keep in, in order to keep everything consistent from year to year, but I think is a bit annoying, actually. It's for the HLMI 1 and not the FAOL 1, we said to assume that scientific progress sort of continues as normal, like there's not some giant impediment to that. Those are roughly the 2 definitions. I guess, actually, sorry, for the full automation of labor question, though, the question is different in that first we describe what full automation of an occupation is, and then we go through a series of questions that ask people about different specific occupations. Like, we give them 4 occupations and ask when they think those might be fully automatable, and then ask them to think of a particularly late occupation and when they think that will be automatable, and then ask them when they think everything will be automatable. So I think this leading them through the process and steps could also lead to a very different answer, and maybe that's what we're seeing.

S3 (16:06) Hey. We'll continue our interview in a moment after a word from our sponsors.

S2 (16:11) That's interesting. I wonder if it would be useful to just open up a like, this is the experience of the the survey. Is there any place where people could go and actually just take it themselves or, like, experience what the and they may not qualify necessarily for your population. But for folks like me, for journalists who wanna understand what actually happened here, is there a version where people can go see the actual step by step?

S0 (16:34) I think there isn't actually. Sorry about that. Sorry. You can you can go and look at all of the questions, but they're all just, like, in a PDF. I think a difficult thing with just seeing what it's like is that there is so much randomization in it that you're going to get 1 particular path through it, which, you know, is still probably informative, but you you would miss most of the questions.

S2 (16:52) Okay. So we've got high level machine intelligence, which is when unaided machines can do all tasks. And then we've got what on the surface seems like a pretty similar concept of full automation of labor, but those do lead people to quite different numbers. Let's talk about how the numbers actually get collected. Again, there's 2 frames here. I mean, I think the the difference is is also pretty interesting.

S0 (17:15) Yeah. If you were to ask a person about, like, when a thing is going to happen and you want them to give you a distribution over time, it's sort of going to look like, well, there's a very low probability of happening tomorrow and a higher probability that it will happen by the next day and so on. So we're trying to somehow get from them a distribution of probabilities increasing over time, and so we wanted to do this by getting 3 different year probability pairs and then drawing a line through them to make a probability distribution that we're guessing is roughly close to theirs. And so the 2 natural ways to do this are to give them years and ask them for the probability that they think that it will happen by that year, or to give them probabilities and ask them in what year that probability will be met. In 2016, we split people in half and gave half of them 1 set of questions and half the other. And it was like, they very consistently give different answers to these 2 things. We also tried it on people from Mechanical Turk, random survey takers for money, and they had pretty similar patterns where, in particular, if you give years and ask for the probabilities in those years, you get a later distribution than if you say, and in what year will there be a 90% probability of this? So given that they're pretty consistently different, we keep on asking them both ways each time, because we don't want to just pick 1 of them and go with it because we know that it's biased. So we want to do something in between the 2 of them. So we ask half each way and then turn them all into probability distributions, and then put the probability distributions back together again and average them.

S2 (18:53) So let me make sure I get this working backward. So the idea is at the end, you wanna be able to say, we have a curve that represents the elite ML researcher community's aggregated sense of how likely these different technology phenomena are to happen over time. And, ideally, that'd a nice smooth curve so we can make sense of it and look up any year and whatever. Now to get there, we have to get everybody's individual take. But to ask people to draw all these curves is pretty tedious, requires an advanced interface. You can do that sort of thing. To a degree, I'm, like, Metaculus, but it's not super lightweight. So instead, you say, well, okay. If we have 3 percentage year pairs, then that's enough. We can fit a gamma distribution. You can maybe tell me why gamma distribution versus a different distribution, But we will fit a a formula to that, and then we can do that for every user. And then we can basically take the average of all of those fitted distributions, and that will become the the final aggregate thing. And then for 1 more layer of complexity, the pairs themselves could be generated in different ways. You could say, in what year where will there be a 50% chance or basically amounts to, like, what's the over under? Right? If you're a gambler, it's like, you give me the year that which you would accept even odds.

S0 (20:22) Exactly.

S2 (20:23) And the flip side of that would be, I give you a year, and you tell me a percentage.

S0 (20:27) That's right.

S2 (20:27) Yeah. And which direction was the bias? I didn't catch which 1 gets the earlier the later.

S0 (20:33) If I ask what odds do you give in 20 years that will get you a later curve, so things happening later. So I guess my guess is that it's something like, for pretty wild things, you're inclined to put low probabilities, perhaps. So if you're given different years, you can just keep on putting low ish probabilities. Whereas if someone says, like, when is it 90%? You still have to give a particular year. Once you're accepting that it is a particular year, unless you decide that this thing is just not going to get that likely, then it's not particularly tempting to put it extremely far out. I don't know if that's right at all. That's how I remember which way it goes because I'm somewhat suspecting that's happening.

S2 (21:14) Yeah. Interesting. Okay. So going back to the gamma distribution for a second, what can you tell me about that? I'm not very statistically sophisticated. 1 question that comes to mind is, does it necessarily reach a 100% at the end? I don't really know anything about what is implied by this particular choice of distribution.

S0 (21:33) Yeah. I'm actually also not sophisticated here, which is 1 reason I have colleagues. Yeah. I'm not sure. I I think that it doesn't have to go to a 100.

S2 (21:42) It's certainly a very thoughtful approach. Reading through the paper, I was like, 2 framings of what it is we're talking about on these extreme super powered relative to today's AI systems of the future. That I think is healthy, just sanity check. It is notable that there's quite a difference between those. The the flipping of the percentages in the years is also really interesting, and it's definitely just a super thoughtful approach to try to find some way to get, like, a continuous distribution from relatively sparse data across people. So I I came away from it feeling like it was better than I would do. That's for sure. And, of course, all these sorts of things are gonna have their artifacts or their weaknesses or their points of question. Would you give any caveats? If somebody used to say, what's the biggest problem with the way that this data has been gathered and and synthesized into this aggregate view, what would jump out to you as the biggest problems with it?

S0 (22:37) Not sure about the biggest problem, but 1 problem I was thinking of then is, like, I feel pretty good about us having asked things in various different ways because I think, in fact, we do see substantial framing effects or different answers for some reason for what seemed like fairly similar questions. So I think that's pretty good to get. But I think that I wasn't thinking about when we were designing this in 2016, because we basically had a very similar survey from time to time, is that if you ask a thing in 4 different ways or something, then it really opens it up to reporting biasedly. Like, you really can choose 1 of the things and advertise it, and even if we don't do that, other people can do that. And so I think I'm pretty keen on not doing that, but I do end up having to fight a constant battle to get the HLMI answers and the full automation of labor answers. I think it's very tempting to just write about the HLMI answers, partly because the full automation of labor answers are so late that maybe they seem more ridiculous or something, especially to people who feel like AI is happening very soon. So it's tempting to just look at HLMI and be like, oh, and it dropped a huge amount. This is all happening fast. When if you look at all of our answers together, it's quite a strong counter signal saying that people are expecting this in quite a long time. And so I think, yeah, all of this kind of different questions about everything makes that quite hard and means that you have to police it yourself, perhaps, and it might make it harder for other people to trust that you're doing that well, which I try to mitigate by making sure that just everything is online somewhere so people can check. In terms of overall worst problems with it, I think just the fact that forecasting the future is quite hard, and these people are not forecasting experts even, and they're often answering it in 16 minutes or something, and it's a lot of questions. The questions are about really complicated, like what will the long term consequences for the world, a pretty complicated thing, be of this technology that we haven't seen at all? I don't expect the answers to be very accurate, but we are having to make decisions about these things. And so I think inaccurate answers, the best guesses that we can get are valuable, and so it's important to hear what AI researchers think about this. I guess I also think that just knowing what different people genuinely think about it is important for coordinating. But, yeah, that's pretty different from these people are probably right. It probably will be the year that they say.

S2 (25:09) Yeah. Well, as we get into results, it'll also, you know, become very clear that there is a wide range of opinions and certainly in the aggregate, a lot of uncertainty, which means that the accuracy question is probably less central anyway because it's not like it's a very tight estimate that we're getting. Right? It's a pretty broad range of opinion.

S0 (25:29) And I think it's often good to to to be clear about what kinds of updates you might make from this. I I think that you can strongly infer that some important things are not ruled out. You might think that these people are experts. They perhaps know that this crazy thought I have is not plausible at all. With some areas that you can get to that sort of conclusion. I think here, the fact that a lot of these AI researchers put some probability on various things is worth paying attention to.

S2 (26:01) Yeah. Well, let's get into it then. The results are are definitely interesting. So I thought maybe we could start a little bit with some general, like, higher level characterizations and then work down to some very particular questions. At a

S1 (26:16) high level, a couple of

S2 (26:17) things that jumped out to me about the distributions, and I'll let you expand on this or add your own commentary as to what has stood out to you the most. But it seemed that and this is pretty consistent, I think, with other things I've seen that there is kind of a left heavy nature to the distributions. Like, the timing of all the tasks. Right? There's 39 different tasks. You know, when will a AI system be able to do this task? And we can start to list some in a in a minute. But across the vast majority of them, I would say, the range of the first half or the second quartile as compared to the third quartile is just much more compressed. So it seems like a lot of people are like, yeah, there's a decent chance that this might happen in the kind of near future. And, obviously, there's varying definitions of near for different levels of difficulty, but there's, like, a pretty decent mass in the kind of near to midterm future for a lot of things. And then there's a really long tail that goes, like, far off into the future. That seems to be a broadly true statement about all the different answers. Is that fair?

S0 (27:22) Yeah. I think that seems right. I guess I haven't thought a huge amount about this, but I wonder to what extent that's sort of what you get if you're predicting any kind of thing where you sort of think it will happen soon, and it hasn't happened yet. So the bit between right now and your kind of 50% it will have happened date is kind of all smooshed up. But then if it hasn't happened by then, there is the rest of eternity to to spread it over somehow.

S2 (27:48) Yeah. I think that's right. In everything I've seen like this, this shape is seemingly a norm. Another way to say it is the median is sooner than the mean. And intuitively, that makes sense because you have a long tail possibility into the future, and obviously, you're not gonna be predicting the past. So fundamental asymmetry kind of always leads to that. But, basically, I I would say for folks who are trying to develop a mental picture, and you can look in the the paper or go look at, like, some of the AI questions on Metaculus, you can see actual curves that very much have this shape. There's, like, a big bump in the relatively short and midterm, and then there's a very long tail that goes on far into the future. Another big observation for me is just that the ranges are super wide. I'd say maybe the headline figure in the paper is 1 that shows the middle 2 quartiles, that is the 20 fifth to the 70 percentile range of expectations of the timing of all 39 specific, you know, abilities that that you ask about. And the range there is often huge. There's pretty high disagreement or in aggregate, you could say basically radical uncertainty about the timing of a lot of these key things. It's not like we're talking this is 12 to 18 years out. It's this is 10 to 100 years out in some cases.

S3 (29:14) Hey. We'll continue our interview in a moment after a word from our sponsors.

S0 (29:19) You said a disagreement. I think there is disagreement. The note that the the lines we're looking at here of when milestones will happen are actually the averages of everyone's distributions. So they don't actually indicate disagreement so much as like within each person, a lot of different things being possible. So if there's a very long line here between 2575%, it's that the distribution you get by averaging everyone's distributions together is very uncertain.

S2 (29:49) Okay. That's a key point. So I think there are some other questions that may better establish disagreement on some key questions, but this is let me just make sure I can repeat this back to you. You're averaging individual distributions. So these ranges reflect the fact that the average individual, so to speak, has provided a very wide range.

S0 (30:12) Yes. I think that's right. I think you could also get this from lots of different individuals providing narrow ranges but at different times when got averaged together. I think you would get a flat distribution, but it at least very well could come from a lot of people having flat ranges. There are figures of, like, for HLMI or full automation of labor, what the overall curve looks like and what some of, like, random subset of the individual curves look like in the background. And so I think you can see there that for those at least, the individual curves are all over the place. Like, they're not in agreement, but also they're quite often very spread out. So, like, each person is quite uncertain often for many of them, but also there's a lot of variation between the people. My guess is that that's also similar for the narrow tasks.

S2 (30:59) Yeah. It's interesting. I really like that visualization. This is figure 3 in the paper you're referring to. Right? I love this kind of graph. It's the bright color line that represents the aggregate and then all the faint color lines behind it. But as I'm looking at it now, for the individual predictions, it is seemingly fair to say that if you have a very steep line, you are expressing high confidence. If you have a very slow upward sloping line, you are expressing very low confidence. It seems like there are some people that are expressing high confidence. Most of those people seem to have high confidence in the relatively shorter term. You don't see too many lines that are super steep that are late in the time range. The lines that are far out that are rising slowly through time, these are, like, pretty high uncertainty. It seems like it's a mix of more confident people who have shorter timelines and less confident people who have just kind of I don't know. But it seems like it's not soon timelines.

S0 (31:58) Yeah. I feel like there are decent number that are, like, I don't know, and soon is, like, about as likely as later. Like, sort of gradually is sort of flat, or are there some that are more, like, you know, flat for a bit and then go up after a while? I mean, I think it would be kind of strange to be like, definitely won't happen for the next hundred years, but they're highly likely to happen in the 10 years after that. But that would be a sort of weird epistemic state to be a negative.

S2 (32:20) I mean, you do see a couple of those in the graph, particularly with the full automation of labor. There are 2 that who, like, shoot up in the 2100 to 2150 range. So, yeah, that is somebody who's saying, when would you give it a 10% chance that there'll be full automation of labor? And this person said, 2120. And then then you said, when when is there a 50 chance? And they said, 2125. And when is there a 90% chance? And they said, 2130. And an octave actually might even look a little tighter than that. But so that is strange, but we don't see much of that. Like, typically, the steep lines are broadly very front loaded where people are like, 10% in a few years, 50% in a few more, and 90% in a few more. You do have a few of these oddballs that sort of have a tight range in the distant future.

S0 (33:08) Or possibly misunderstood the question or something. We tried to filter out people who are clearly misunderstanding it, but I think there are some confusing cases.

S2 (33:17) Broadly left leaning, that's kind of an inherent function of this sort of forecasting dynamic. Generally, pretty broad distributions and the mix of personas where some people have higher confidence in shorter timelines, some people have very broader timelines and and lower confidence, and a few that sort of buck the trend and do something that may indicate that they misunderstood the question. The next big finding is and this is, like, from 1 vintage of the survey to the next that timelines are probably coming in. So there was a 2016 version of this with a relatively small sample comparatively, a 2022, which had I don't know how many 100, but, like, a pretty good sample size, like, the kind of thing that they would do a a national presidential poll with. And then this 1 is, like, several times even bigger than that. You wanna summarize the general pulling in of timelines?

S0 (34:12) I think the really notable thing is that between 2022 and 2023 surveys, there was a big drop in these roughly human level performance questions that HLMI and full automation of labor, where, I guess, like, the HLMI 1 dropped by about a decade and the full automation of labor 1 dropped by, you know, was it 4 or 5 decades? And I think that's pretty notable given that between 2016 and 2022, HLMI, I think, changed by about a year, so it's not just fluttering all over the place. And I guess for the narrow tasks, you also see a general drop. I think on average, it's by about a year there, where there's some dropping by a lot and even some going the other way, but more of them dropping. This is like the year dropping, not like the number of years until the thing happens dropping, which you would expect to get 1 year less by a year passing.

S2 (35:04) Yeah. So the actual specific years that are predicted are coming in. Yeah. It's worth just reading some of these tasks. 39 different tasks, which each have a short name and then a full description that the survey takers get to read. I'll just give a couple of them. Short name, physically install wiring in a house. Full description, given a 1 sentence description of the task and given the same information you would give a human to perform this task, such as information about the house, physically install the electrical wiring in a new home without more input from humans. So that's obviously a pretty challenging task for today's AI systems. Also, I think that definition is, like, pretty representative of the 39 as a whole where it's broadly, here's the task. We are interested in, can you delegate that task to a AI system with basically the same ease of delegation as you would delegate to a human? The state of AI task automation today is quite different from that. I can get GPT-four to do a lot of tasks, but it's definitely harder than in in many cases in the setup than it is to ask a teammate to do that same task for multiple reasons, including they don't have a lot of context. They're not great with context, just broadly speaking, for multiple different reasons. So I'm not really sure what to do with that as I try to understand these results more broadly, but I do notice that there is a a pattern in the questions that's like, can you treat the AI system roughly as a human? Give them kind of terse instructions, a little bit of context. Here's the blueprint. Go. Whereas today, what I typically tell people is we can probably save you 90% time and money with task automation, but you're gonna put the 10% in the front to set up a system, gather the context, do validation, workshop your prompt, maybe fine tune a model, whatever. It's definitely not nearly as easy to do that delegation to the AI, but you can still kind of get there.

S0 (37:10) I guess I would have thought that at the moment that it wouldn't be largely upfront. It would be, like, with a lot of, like, input along the way. Like, AI at the moment isn't able to act like an autonomous agent that's trying to do something for a long period of time in a useful way without you giving more input or redirecting it. Like, you're the 1 kind of directing the task, at least in my experience of doing tasks. So I've seen people use it for lots of different things. But I don't It's like, I'm writing a thing. I'm like, I wonder, you know, other examples of this thing. I'll ask it. And I'm sort of like directing small bits to it.

S2 (37:44) Yeah. I think this is all it was a very important practical distinction in AI task assistance or task automation. The flow that I'm describing there is 1 where there is some scale at which the task is meant to be completed, and the goal is that you would get the AI performance to a level where you don't have to review every single AI task execution once you're satisfied with, you know, the level of performance. So I usually distinguish between ad hoc real time copilot style usage, which is, you know, I'm writing something. Can you help me edit this paragraph or whatever? But you're not doing that task at scale. And on the contrary, the classic tasks that almost everybody has certainly in business. Would you like to personalize correspondence at scale? Yes. I'd love to. However, who has time to do that? Well, AI does. Right? Now can we take your database and understand what kind of personalization matters and set this up and take the first 100 results back to your copywriter and make sure that it's working on that level and whatever? Every different situation has different kind of requirements. But, also, a big part of that is just conceiving how are you gonna break the tasks down in the first place. Right? That is a big part of it. So I guess maybe that's really the the key distinction between kind of what in practice is done today and the paradigm that you're sketching out in the survey questions, it's like, who is responsible for breaking down the task into these, like, subtasks, each of which could be developed and validated, whatever. In the survey questions, that's on the AI. It's not the the human, generally speaking, is not responsible for getting super granular and really controlling. It's supposed to be high level delegation of the sort you would give to a a teammate, not the sort that you would give to GPT-four today.

S0 (39:36) I think it varies between tasks, but at least for some of them, they're getting pretty far in that direction. I guess my model of such things is probably kind of fractal issue. Like, you can have some skills that allow you to do a second's worth of useful work before someone has to redirect it. Or if you have a bit more skill or some other kind of skill, then maybe you could put together 3 different pieces, but still, like, someone above you has to be like, alright, now is the time for that, and so on. Like, it might have been that in the past, I could use a thesaurus, and it's like, alright, I know that I I want this particular task done. I can check. I'm like, now chat GPT can do several things like that at once without me telling it what I want. Like I say, I want you to write this letter or something, and it can, you know, know that it should think about different words for this place and know that it should think about how to be polite at this place and know it should think about what are the things I was going to list in the letter or something. And so it's putting together several different things, but still it's not able to be, like, at a higher level directing what happens with this letter writing. For instance, I still need to be like, this is gonna need some kind of quality check, and someone other than the AI is gonna have to do that. I'm gonna have to figure out who to send it to, that sort of thing. And so I think I'm imagining like, my picture of all of this is that you gradually grow toward these things as more and more stuff is able to be done by 1 system.

S2 (40:54) Okay. Cool. I'll read a couple others just to give a little bit more concrete color. These are not easy. Right? The next 1 is fine tune LLM. Given a 1 sentence description of the task, download and fine tune an existing open source LLM without more input from humans, The fine tune must improve the performance of the LLM on some predetermined benchmark metric. That's an interesting 1 because that's 1 1 of a handful of thresholds that I definitely watch out for. Another 1 is replicate ML paper. Given a study published at a leading machine learning conference, replicate the study without more input from humans. The replication must meet the standards of the ML reproducibility challenge. It's a link to the the definition of that. There are a few in here that are certainly relatively easy, although most of them, I would say, are pretty hard.

S0 (41:44) I think arguably some of them have already been done.

S2 (41:46) Yeah. I wanted to actually ask about that 1 too. Now this sort of both validates and calls into question some of the data. In my view, probably the easiest and in in the aggregate view of the respondents, the easiest task, that is the 1 that is most likely to happen soonest, is write readable Python code for algorithms like quick sort from specs and examples. Full version, write concise, efficient human readable Python code to implement simple algorithms like quick sort. That is the system should write code that sorts a list rather than just being able to sort lists. So write code to sort a list. Suppose the system is given only a specification of what counts as a sorted list and several examples of lists undergoing sorting by quick sort. That 1, I went to ChatGPT and asked it to do a quick sort in Python for me, and I'm pretty sure it nailed it. And so you could say, well, that's, like, in accordance with the data and that the results there were the soonest. But it still seems like a full 25% of people said that that wouldn't happen until 2029 plus. And that is 1 area where I was like, Are, like, a quarter of people, like, not aware of ChatGPT, or are they, like, interpreting this differently and sort of generalizing to, like, other algorithms? I mean, if I'm trying to, like, steel man, the the case here would be like, well, QuickSort's in the training data. Maybe this would could be understood as, like, for new algorithms that you have examples of that are not in the training data. Do have any sort of way of understanding how that result makes sense?

S0 (43:18) Not fully, but I have some thoughts. 1 is, I think you're misunderstanding the it's not that 25% of people thought that. It's more it's that everyone together, average together, thinks that there's like a 50% chance of this happening, what is it, like, a couple of years in the future or something. In my experience, like, demonstrating that 1 of these things has properly been done is surprisingly tricky, especially if maybe that 1 is pretty easy, but yeah, if it's like, can it consistently do this across the board? Can it do it for other things? Basically the same as quicksort. I didn't think in the question we really said what fraction of the time it has to succeed at it or something. It might be like, in my use of chat GPT, say, my experience is that it's often great at things and then often terrible at things that I would've thought it was great at. Last I checked, it wasn't very good at counting things. It was like, how many ones is this in a row? It's like not very good at that. And so I could imagine, like, without actually going and trying the thing right now while you're doing a survey, you're like, well, this seems like the kind of thing it can probably do. So probably sometime between right now and in a couple of years or something. I think also the fact that we included it on the survey, I think they might take as evidence that it can't be done right now. We decided not to take any of them off since 2016, partly because it's just so complicated to figure out if they've actually been done or not. So we decided to not have any other opinion on that and just include them all, and maybe take them off once the respondents start to actually say this has constantly happened instead of like, I don't know. Maybe 5 years. But I think that might be confusing people.

S2 (44:50) Yeah. Isn't that an option? There's not an option today, right, to say this already exists?

S0 (44:55) There isn't. So, yeah, quite plausible we should add that, and that would make things less confusing.

S2 (45:00) Zooming out again, on 35 of the 39 tasks, we have a 50% chance that it will happen in 10 years or less. So here's the 4 just to calibrate on the ones that were greater than 10 years. 1 was the installation of the electrical wires that was estimated 50% at 17 years. The ML paper, there's a replication 1 that I read that was estimated at 12 years, 50% chance of at 12 years. And then there's also a research and write. In other words, do your own ML paper from scratch. That 1 is 19 years. And then the furthest ones out were around math. Interestingly, there's some interesting proof points there as well of late, but proved mathematical theorems that are publishable in top mathematics journals today, 22 years out, and solve long standing unsolved problems in mathematics such as the millennium prize problem, 27 years out. So these are the hardest ones with the longest timelines, things like Python, things like playing angry birds at a human level. The large majority of these things are under 10 years, 50% chance. I would say another kind of observation, I wonder how you would react to this is it seems like the timelines here, while they have come in and while, like, most of these things are more likely than not to happen inside of 10 years per the aggregate judgment, it seems like the timelines here are still longer than the guidance that we're getting from the heads of leading labs. Like, I think if Sam Altman, if Dario, if Demis and Shane Legg took your survey, I think their numbers would be on the shorter end of the aggregate.

S0 (46:42) That would be my guess. I think also just from many sort of people working in AI who I know here in the Bay Area.

S2 (46:48) Yeah. I don't think we have 3 year percentage pairs from the the heads of the leading labs.

S0 (46:55) I can't tell if we do or not.

S2 (46:57) Well, here's things that I've seen recently. Sam has said AGI is coming soon, but it might not be as big a deal as you think. He also said at a Y Combinator event that startup founders now need to build with AGI in mind, which is certainly an interesting app design challenge. Anthropic, we don't tend to hear quite the same thing too often, but there was the, I think, credibly sourced pitch deck that they had where they said that in 2526, the leading model developers might get so far ahead of others that nobody will be able to catch up because they'll have their own kind of feedback loops where they can use their current models to train the next ones. That certainly sounds like a scenario where things are happening faster than the aggregates from the survey. And Shane Legg from DeepMind recently said that he's basically had, like, a 2029 median timeline for, like, 15 years.

S0 (47:49) I think it's interesting to note that having had that timeline for a while and similarly, perhaps for some of these other people, I think you could wonder, like, hearing very optimistic timelines or, well, optimistic or pessimistic depending on how you feel about this, but, like, very soon timelines. You might wonder whether that's from seeing something right now that suggests that it will be very soon versus being kind of selection effect where people who think that AGI is soon work on AGI. And I think where we observe that people already had these views some time ago, that support for the selection effect explanation of the the difference in opinion there.

S2 (48:23) Yeah. So for comparison, the survey gives a 10% chance of high level machine intelligence by 2027. And for a 50% chance, you have to go out to 2047. So that's pulled in significantly from just 1 year previously. In 2022, it was 2060 that you had the 50% chance. So from 2060 to to 2047, 13 years in, but that's still quite a bit farther than certainly what we're hearing from all the the leading labs. Like, what the survey says is sort of a 10% chance. It seems like the leading lab heads think it's, like, more likely than not on a similar time frame.

S0 (49:06) Images by impression. Yeah.

S2 (49:08) Yeah. So who do we believe? Obviously, very hard to say. Yeah. 1 thing that you've done that I think is pretty interesting is and I might even actually spend a little more time on this is you've published all the results in cleaned up anonymized form so folks can go do their own data analysis on it. So would there be anything stopping me from taking the results and saying, I'm going to go filter out anybody who thinks that the Python thing is not close and rerun the analysis with those people filtered out? It is hard for me to put a mental model on I don't know how many people I filter this way, but to me, there's gotta be a a fraction, like, meaningful fraction that might be skewing the tails where if it's like, you published in a leading conference in 2022, but you are saying that an AI won't be able to write a quick start for a long time. Like, I am kind of confused about that. Maybe I should just filter you. I could do that, right, with the data that you've published?

S0 (50:02) I thoroughly encourage that. I would love to see more people use it. And I think there are a lot of interesting things you could ask about, and it would really only touch the surface with giving the basic answers to each question and some amount of how does this relate to demographics. Yeah, I think how do the answers relate to each other? There's a lot of interesting things there. I think if you filtered out the people who said that quick thought wasn't possible for at least 15 years or something, I'm not actually sure what you would be getting there. I think you might be filtering out a mixture of people who didn't understand the question or just made an error on that 1 or who have some sort of complicated philosophical take. Like, it's not doing quicksort. It's doing matrix multiplication or something. It's not doing

S2 (50:44) real quicksort. It's just doing sort of stochastic period imitation of quicksort.

S0 (50:49) I I don't know. And then I guess if you filter out those people, I don't know if they're more likely to be wrong overall about other things. Yeah. But I I feel like they're more likely to have thought about things, probably. We did actually just ask people how much they've thought about different things, so I think a very natural thing to do would be to see what just the people who said they thought the most about things think, though I guess that's also tricky because of the selection effects with people who are more concerned about things, thinking about them more. So I think in fact, the people who thought more do seem to be more concerned, but I'm not sure what we should actually take away from that.

S2 (51:21) I think we'll return to that correlation question in a second. Just carrying on to a few more headlines. It was definitely striking to me that people are expecting the unexpected. Here's a quote. A large majority of participants thought state of the art AI systems in 20 years would be likely or very likely to find unexpected ways to achieve goals. More than 80% of people expect that in 20 years. Be able to talk like a human expert on most topics. Again, more than 80 expect that. And frequently behave in ways that are surprising to humans. Just under 70% of people expect that to be the reality for AI systems 20 years from now. I have to say I probably agree with all those things, but that is definitely a striking result. Right? That people are expecting ongoing surprises from AI systems as their default. That seems like the closest thing probably to consensus in this survey is that everybody expects these things to be unwieldy and surprising even as they become, like, quite a bit more powerful over a 20 year period.

S0 (52:25) Yeah. That does seem like 1 of the more things, at least. Yeah. I I guess I don't know, like, how sinister to hear some of these things as, like, finding unexpected ways to achieve goals. At some level, that's just, like, what you expect if you are getting someone else to achieve goals for you. You're not going to figure out all the details of how to achieve it. They're going to do it, and you'll be like, Oh, nice. Figured out a way to do that, versus being quite surprised and being like, Well, that was a norm violation. I thought I should put up more barriers to prevent you doing that, but I bet you did. Yeah. Similarly for the sort of frequently behave in ways that are surprising to humans. I feel like that could be more terrifying or more just what you expect. Yeah, I would have liked it if we'd been clearer with those questions a bit.

S2 (53:09) Yeah. Maybe there's a way to tease that out a little bit through correlating with the next headline result, which is, to my eye, the group is only slightly more optimistic than neutral. Basically, you've got a few of these kind of classic questions where you're like, 1 to 5, like, extremely bad, bad, neutral, good, or very good, and just ask people how ugly do you think this is to be in these various buckets.

S0 (53:34) To be clear, this is, like, how good or bad is the long term future as a result of HLMI in particular. Yeah. Each person puts a 100% between the 5 buckets.

S2 (53:44) And there I would say this is, like, the clearest sign of radical uncertainty here because all 5 of the buckets have significant percentages, and they are skewed slightly positively.

S0 (53:56) It looks to me like a fair amount of agreement on extreme uncertainty. There's a good bulk of this graph that is people putting a chunk of probability in each of the 5 buckets and just a little bit at each end, where people are either very confident in good or very confident in bad. And this is a question that everyone in the survey got, and everyone had to add to 100%, and they had to answer it in order to get to the next question. So, basically, everyone answered this. Yeah. Each column is 1 person.

S2 (54:23) Yeah. Okay. And this is figure 10 in the paper. This is definitely a cool visualization as well. It's a sorted list where each person gets a vertical pixel, and then you can see how each person distributed their expectation from extremely good to extremely bad. And because it's sorted, you have an extremely good region on the 1 end and an extremely bad region on the other end. And in the middle, you can see that, yeah, like, a lot of people have given nontrivial weight to all 5 of the buckets. And, like, the middle sort of 2 thirds of people probably have agreement on radical uncertainty. And then there's, like, a maybe a sixth on either end that are sort of the optimists and the pessimists.

S0 (55:04) Even, like, most of those are, like, pretty uncertain. Even if you look at the the big black chunk of pessimists at the right hand side, still, like, most of their area is not extremely bad. It's a decent chunk of extremely good in there.

S2 (55:17) Yeah. You got a few bimodalists. I I don't know if I quite do call myself a bimodalist. Sometimes I do, but I'm not quite ready to stake my identity on it. But there is definitely a band there where you only see the 2 colors, and it's the extremely bad and the extremely good.

S0 (55:32) I think I'm also seeing it like a a 2080 band like that. But if you said extremely bad, it's an extremely good. Yeah. Okay. I I like seeing that, like, how polarized this isn't. I think this kind of thing can feel polarized if people are talking about doers and so on, but it's, I think, very not.

S2 (55:51) Yeah. Totally. Definitely suggest a very big middle of people that are just very uncertain as to what to expect. And those people are probably, for many obvious reasons, not the most vocal online. But I think a huge part of the value of this work is demonstrating that there is no single consensus. There is no dominant view. There's no single number that we can put on this. But really, for me, the headline is just there is radical uncertainty expressed, like, any number of different ways by the community at large.

S0 (56:22) Radical uncertainty is in some sense a consensus about something. Like like, I think here, you could say there is more or less a consensus that there is, like, non negligible probability of extremely bad outcomes. And from a sort of action perspective, if the question is yes or no, is this a risk worth paying attention to, then it ends up being sort of like a consensus for yes, worth paying attention to, because uncertainty is like, yeah, there's some chance of it, and you'd actually need to be pretty confident on something to say, no. Not an issue.

S2 (56:52) Yeah. I think that's a good point. The headline that I've seen from the coverage of the survey broadly has largely focused on what you might call the p doom point estimate. And I've seen this most often reported as a majority, and it's a a slight majority, 51% believe that there's at least a 10% chance that AI could lead to human extinction or similarly severe disempowerment.

S0 (57:18) 2 things. I think the thing that I've most heard at least is the median 5%. But, also, I think that 10% would be wrong here because when we ask this question about value that we're just talking about, then we ask 3 other questions that were quite similar about human extinction in particular. So table 2 shows the 3 different questions that we asked that are all quite similar about human extinction or similarly permanent and severe disempowerment of the human species. 1 of them is just straight up asking about that. 1 is asking about that as a result of inability to control future advanced AI systems, and 1 is asking about it within the next hundred years in particular. And so the medians for those were, like, 5%, 10%, 5%, and I guess the means were quite a bit higher.

S2 (57:59) So that headline is cherry picking the highest median of those 3. It's interesting too. This does show some of the limitations of both surveys and interpreting surveys. I think it's very interesting framing, but when you really start to stare at it, you're like, well, there's definitely a sort of conjunction fallacy at work here. Right? Like, the first question says, what probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species? Median answer, 5%. The next 1 is essentially the same question, but with an added detail of inability to control. And as far as I'm seeing here, every other word is the same, and the percentage chance doubles. Right? So if you're, like, logically consistent entity, presumably, the second 1 would be, like, lower. Right?

S0 (58:45) I think it's not necessarily coming from the conjunction fallacy, though I think it may well be. But things to note here are, like, people are randomized into each of them, so it's not like the same person answering them. So it could just be some variation from them being different people. But I think often a lot of people put 5% and a lot of people put 10%. So whether the median is 5 or 10% is down to like exactly how many put. Where the middle person lands, there are like a bunch of people putting 5%, a bunch of people putting 10%, and exactly where the middle is changes it from 5 to 10%. But it seems wrong to be like, oh, they doubled it. The distribution of people thinking things is similar. Last year, I remember 48% of people said 10% extremely bad. So it's like, if it got up to 50%, then the median would be 10%. But because it was 48%, the median is 5%.

S2 (59:31) Gotcha. So these are bucketed answers. In this case, we're not doing a distribution or a free response point.

S0 (59:37) Right. Or or so we're asking each person, what is the chance of this? And in fact, people just never put numbers between 510%. They're sort of rounding it themselves or not never, but, like, quite rarely. So I think in practice for these questions, the median jumps fairly easily between 510% rather than hitting intermediate values. I do still think that probably some amount of consumption fallacy is going on here, partly just because we saw the same pattern last year.

S2 (1:00:05) Again, to attempt to summarize the the headline p doom point estimate is 5 to 10% median estimate with certainly, again, a kind of left heavy distribution and the mean being higher than the median, and that's presumably driven by a minority of people who have high estimates. You can see that again in any other figure as well. Well, that's definitely something. That's pretty consistent, I would say, with my own. I don't spend a lot of time trying to narrow down my p doom. I thought Dennis did a pretty good job answering this question. He did the New York Times hard fork podcast recently, and they asked him what his p doom is. And he was like, it can be a subtle distinction, and people wanna collapse the distinction. But he's like, we really just have no idea. It's not like we have a process that we have established that has any number attached to it that we're now gonna execute and find out. On the contrary, we really have no idea what the nature of the process is that we're going through. And so it's just, like, super unknown. I typically say either 5 to 95% or 10 to 90%, something like that. And, certainly, the consensus, at least the low end of that range, is, like, not objectionable to the the large majority of people.

S0 (1:01:18) Pretty in line with them. I disagree with you, or I I agree it's very unknown, but I think that's what probabilities are for. So I'm a big fan about putting numbers on it anyway. And I think that even if you can't do anything except guess a number, it at least allows you to compare to other numbers you're guessing or, like, make consistent decisions over time. I I practice guessing numbers about lots of things, and then I can check how how well that's going. And so whether to listen to the numbers that I make up about this is very uncertain, but I think it's worth trying.

S2 (1:01:45) Yeah. I I certainly find a lot of value in this work just in grounding the idea that this is clearly not a fringe question. It is clearly worth thinking hard about. The Overton window is wide open, and it should be wide open if the elite researchers in the field think that there's a 5 to 10% chance of extinction level bad outcome. So I think that's, like, definitely really true. I guess the way I tend to think about these numbers is what can we shift the true probability to as opposed to how can I become more accurate in my estimate today? And I think both of those are worthy pursuits, but especially for somebody in Dennis' position, he's like, what I need to do is create an agenda that collapses the uncertainty and hopefully moves the distribution.

S0 (1:02:29) I agree that, like, trying to put numbers on such things does potentially get you into a headspace where you feel like you're not controlling the number, and I do think that's bad.

S2 (1:02:37) Yeah. Because we're not, like, about to spin the roulette wheel. Right? Like, there is no roulette wheel that we can, like, spin now with these numbers. I think that's the main thing that I think is worth keeping in mind as a caveat. Right? We're still building the wheel, and we do we have a lot of uncertainty as to what the roulette wheel looks like. And the more concretely it comes into focus, then presumably, distributions will start to narrow. Although, maybe not. It certainly wouldn't shock me at this point if we have just super high level of uncertainty and disagreement right up until a phase change hits, then it's like, well, we just found out. And it wasn't ever very clear until relatively shortly before it happened. Again, it's just another dimension of pretty radical uncertainty.

S0 (1:03:17) I think when you're talking about, like, the probability of a thing conditional on us following some particular path, it's a reasonable way to do it. Like, not, you know, p doom very high. We're going to die or something. Well, like, if we, the humans, are building this particular thing, I think that particular thing will be quite bad. I mean, we should build some other thing instead of maybe a more, like, proactive way of thinking about it.

S2 (1:03:38) Unfortunately, 1 kind of downer result is interpretability breakthroughs are not expected in the short term. Most respondents, 80%, considered it unlikely or very unlikely that users would be able to understand the true reasons behind AI decisions in 2028. Pretty logically, I would say there's a strong agreement that AI safety broadly should be more prioritized than it is today. Very few people saying it should be less prioritized than it is today. Also, should AI be going faster or slower? Again, only 5% of people said it should be going much slower than it is today, but then a healthy relatively even 30% on somewhat slower, 27% on current, 23% somewhat faster, and 15.6% much faster. So am I right to read that as high agreement that we're not expecting interpretability breakthroughs? High agreement that more AI safety work would be good, but high disagreement as to whether we should be slowing down or going faster.

S0 (1:04:41) I think that seems right to me. I think 1 complication interpreting that last 1 is that the the central option, I think, is, like, relative to the current speed. So the most natural literal interpretation is, like, whatever the current speed is good. But, like, if things were to accelerate, you would want them to go slower, and you might think they would naturally accelerate. My guess is that people weren't thinking about it for that long, and what they actually mean is more like that the current trajectory is good, which is perhaps accelerating. But, yeah, I think before you read too much into that 1, it would be nice to ask people some related questions again or something and get a bit

S2 (1:05:14) more clear. So let's summarize this whole thing. I guess, high level summaries, People do not dismiss existential risk. On the contrary, point estimate is, like, 5 to 10%. People have pretty wide uncertainty about exactly when AI will be able to do various things, though those estimates are broadly coming in.

S0 (1:05:39) Also, a lot of them are relatively soon.

S2 (1:05:41) More than 80% of the tasks had a 50% estimate of happening in less than 10 years. So no no breakthroughs expected in interpretability, more emphasis on AI safety, and disagreement or confusion on faster or slower.

S0 (1:05:57) It all seems right.

S2 (1:05:58) As a single headline, we really don't have a a confident take. As a community, I should shouldn't even say we because I have not published in these conferences. I wouldn't be qualified to complete the survey. But if I'll flatter myself to be included in the community, then we, as a machine learning research community, do not have a confident view of what's gonna happen. On the contrary, we have a a pretty radically uncertain view of what is gonna happen. How long it's gonna take to get to different things, whether it's gonna be good or bad, whether we're gonna be able to control systems or even whether we'll be here at some point in the future. Like, all of those are very live questions with nontrivial weight on all the options.

S0 (1:06:38) 1 question we didn't talk about where I think it was interesting that there was more consensus perhaps is the 1 about different scenarios and how much concern they warranted. Not clear that they are concerned, but they think the thing deserves concern from society. The very top 1 there was making it easy to spread false information, e. G. Deep fakes. I think more than 80% of people thought it was worth either substantial concern or extreme concern. That's an unusual degree of agreement.

S2 (1:07:04) Yeah. So these scenarios, just to read a couple of them, AI makes it easy to spread false information, e g deep fakes. AI systems manipulate large scale public opinion trends. Authoritarian rulers use AI to control their population. Other comes in as the the fourth most concerning scenario. AI systems worsen economic inequality by disproportionately benefiting certain individuals. AI lets dangerous groups make powerful tools, e g engineered viruses. Bias in AI systems makes unjust situations worse. E g AI systems learn to discriminate by gender or race in hiring processes. All of those that I just read had more than 60 of people saying they are either substantially concerned or extremely concerned. And for all of those, fewer than 10% of people said no concern.

S0 (1:07:50) It's either not really about what will happen, but what could happen enough that it's concerning. In some sense, for that to be the only thing that's really clear consensus about is maybe supporting your hypothesis. They don't they didn't know what's going to happen.

S2 (1:08:02) Yeah. None of these are dismissed. That's for sure. Okay. Cool. Well, I love the work because it's just so methodical. As we've established through a couple of my misunderstandings, it's pretty nuanced. It's pretty granular, and they can pick it apart in a lot of different ways. But it does seem that it's very well established by this result that there is a lot of uncertainty about what we're in for. There's a lot of uncertainty about even the highest level questions of whether it's net good or net bad, whether we are going to survive it or not. It is not like a doomer community, but it's not a community that dismisses the tail risks either. And so with that, we head into a very cloudy future, I think. For the future, I assume you guys are planning to run this again. A few things that came came to mind for me that I could throw at you. But before I do that, what are you considering doing differently in the future, or what do think are the sort of natural evolutions from here?

S0 (1:08:53) I am reasonably likely to run it again. I think the most natural thing to do is to just run it identically to the past, because it's nice to be able to compare these things. This time we added various questions that we hadn't had before, like this 1 about the concerns and the other 1 about the different traits that AI systems have in the future. But it's like a fair bit of effort to add the questions. And so the project can be a lot more contained perhaps if you just have a survey and send it out every now and again. And so it's somewhat tempting to basically do that. I think also, it's quite hard to cram more questions in here and still have it reasonably short, give each question to fewer people and randomize and so on. It's very tempting if we want to do more surveying, run more surveys. I've been thinking a bit about having something about policy, but I haven't thought about it that much. I guess people send us various questions they're interested in.

S2 (1:09:43) I guess if if there was 1 thing I could ask for, and it's because multiple people have asked me, people don't know how to understand affordances. There is a ton of investment over the last year that has gone into the application layer, connecting models to databases, of course, for rag type implementations, connecting them to APIs and other tools. Now we've got, like, the memory scratch pad. A great paper on this is cognitive architectures for language agents where they survey the full literature and try to put a taxonomy on all these things. But I do think it's pretty interesting to consider what happens when with all this scaffolding, as it's often called in the application world or affordances, as it's otherwise called, when all that is built and it already exists and it's like working okay, you know, or even maybe pretty good with a current model, What then happens when a core model upgrade happens? And all of a sudden, it's like a drop in replacement to a structure that has really been, like, built up to try to compensate for all the weaknesses, and now maybe those weaknesses drop by, like, an order of magnitude. To what degree does that create a situation where you essentially are, like, flipping a switch and a lot of things go from sort of working to working really well, and how does that play out? I do see a lot of potential for that, but people have been asking me for, well, what are scaling laws for scaffolding? And I I just find I really don't know. I I have this kind of broad mental picture of a lot of work has gone into it, and a model upgrade is gonna make a lot of existing systems that don't work that great yet work quite well. I'm not even sure what questions I would ask, but I would definitely love to get a community sense for the relationship between core model power and affordances or scaffolding infrastructure that's already out there.

S0 (1:11:31) Where affordances here mean something similar to scaffolding.

S2 (1:11:34) Yeah. Basically synonyms. Yeah. What tools do you have access to? What information do you have access to? It's all the things that are outside of the weights that the model can use at in for its time.

S0 (1:11:44) You wanna predict what will happen when the new model comes out with the scaffolding.

S2 (1:11:50) Yeah. It's there's a latent potential there. It is a tricky 1. But, yeah, the key thing that I'm wondering about is to what degree should we be expecting a step change in the actual AI impacts when a new model gets plugged into existing enabling infrastructure? Because that seems like a a fundamental difference between AI and almost every other technology is that all of the distribution infrastructure already exists, and the complements are getting built now. And people are very aggressively and eagerly building all these compliments in anticipation of the next upgrade. And they're tolerant of the fact that, like, it doesn't really work now because they fully expect that there's gonna be an upgrade that's gonna make it work. But then that has potentially very unpredictable consequences when all of a sudden everything turns on at once.

S0 (1:12:42) I don't know a lot about the details of this kind of world. I would have thought that there are things a lot like that in other kind of technologies that get updated and have a widespread reach.

S2 (1:12:53) Yeah. I think so. Although in the AI application world today, there's a ton of building that is going into essentially the kind of ad hoc delegation that a lot of the survey questions anticipate as well. People are like, would be sweet is if I didn't have to ever write a SQL query anymore, and I could just ask my SQL agent to look at my database schema and figure out what's what and then write the thing. And the same thing for web scraping and the same thing for answering phone calls and making phone calls, and it just goes on and on. Right? There's research assistants and programming assistants. So it just goes to every corner that you could find a agent type product, legal work, medical research, diagnosis research for yourself. And they're all like, well, GPT-four is not quite there, but it's good enough that I can build a system and track how well it's working. And even if it's not working that well, then I know that there's like a countdown clock on to the next thing. And that's where I feel like we're building a new capability overhang. Right? There's been the notion for a long time of, well, if the chips are all out there and nobody trains the models, then someday somebody could come along and have this, like, massive advance. But here, it's like, on the other end, it's all of the compliments and guardrails and access to data and access to runtimes and the loops and the ability to cache skills. And all of these things are getting built, but the model isn't that good at using any of them yet. And so if it goes from 1 level to another, how does that kind of cascade through the overall system? And do we suddenly move from an era where nothing is really working super well? So everybody right now can assume that they're essentially acting in isolation. Most everybody's worldview right now is like, the world is the world. I'm applying AI here. Nothing else is changing. That's the implicit assumption.

S0 (1:14:38) But you're thinking, like, that's happening under the.

S2 (1:14:41) Yeah. Everybody's doing that, and it is so far true that, like, nothing else is really changing. But I think there's at least potential for a model upgrade to suddenly put us in a different regime where it's like, now my thing is working much better. Now I'm actually gonna go use it a lot more and send it out potentially semi autonomously to do stuff. But at the same time, everybody else's autonomous systems are also getting unleashed. And now we have all these, like, autonomous systems going off into the world at the same time and potentially encountering each other, and we're definitely just not at all prepared for or even thinking about what that might look like. It it could be not a huge deal if the next upgrade isn't that big, but my sense and this goes back to the Sam Altman comment of you should be building with AGI in mind. Like, my sense is that there is at least 1 more round of, like, substantial upgrades. And so do we have this kind of phase change that happens suddenly? It could happen before the next survey can get run.

S0 (1:15:32) Yeah. Often technologies change gradually even if there was a kind of big insight or something. But that's often because you're, like, building stuff to make use of it afterwards, and it takes a while. And so you're saying, maybe if we build all this stuff to take advantage of it ahead of time, because we already knew it was coming and we had this lesser version of it to play around with, that it could be much more discontinuous than most technologies when that bit gets swapped out. Pretty interesting theory. I guess I wonder how much we've seen that with past upgrades like this. I think that my explanation here for why AI would be different to, like, any other technology in this regard would be just that it's, like, quite general. So you are using it across the board, and, you know, there is the potential to suddenly upgrade a thing everywhere. But then you might expect that also to apply to the change between GPT-three and GPT-four, GPT-two and GPT-three. Maybe those are not useful enough to be building stuff around yet.

S2 (1:16:22) That's my sense. I feel like only with ChatGPT 3.5, certainly with 4, but probably not with 3. And that whole time frame is also, like, fairly condensed. It was only, like, 3 and a half from 3.5 first release to 4 first release, and 3.5 in ChatGPT dropped it basically the same time. So it's, like, 3 and a half months from ChatGPT to GD 4. Yeah. I I just don't think before that there was really much. The gold rush or the sort of everybody goes stake out their place in the AI app here seems to have been a last 12 to 14 months phenomenon at most. And, like, certainly, there was stuff going on before that, but I would say there's been a phase change of how much activity there has been.

S0 (1:17:04) You're saying this would be an interesting thing to have a survey about?

S2 (1:17:07) Yeah. I mean, I I'm very uncertain about it. Quality people have have asked me about it. So I have the strong sense that among a thought leading population that's somewhat in some cases adjacent to, like, policy decision making, there is a a lot of uncertainty and a lot of questioning around that particular question. To what degree should we expect discontinuity due to the plugging in of new models to existing infrastructure. You know, maybe we could today use that as a first draft. But, yeah, that's been the number 1 assignment that I've gotten from some of my highest value correspondence when I'm struggling to model it. So that's my case for a bonus question next year.

S0 (1:17:46) Okay. Thank you.

S2 (1:17:47) I guess other areas that could be of interest, would you look at, like, military or US China chip restrictions? Is Sam gonna get his $7,000,000,000,000? Maybe that's, like, outside the scope of what ML elite publishing folks should be consulted on.

S0 (1:18:03) There's also, like, what do you do with those answers? What does someone usefully learn from these things or do differently as a result?

S2 (1:18:11) I Well, escalate or deescalate with China may be 1 key question. I don't know that our executive decision making is necessarily going to be a super evidence based in the near term depending on who's in charge, but I do feel like that's another question where I have dramatic uncertainty.

S0 (1:18:28) About what ought to happen or what will happen?

S2 (1:18:30) So is the chip ban going to work? Will it meaningfully slow, or will will there be, like, a multiyear gap between US and Chinese frontier models as a result of the chip ban. I am hearing confident takes on both sides of that issue. I've had people tell me who I think are very smart. Oh, it's definitely working. They're not gonna be able to keep up. And then I look at certain models, and I managed to go as far as trying Ernie 4, and it seemed pretty good. At a couple of queries head to head with GPT-four, it wasn't, like, obviously dramatically inferior. In fact, it seemed pretty comparable. And there's a lot of Chinese speakers at these conferences, obviously, too. So that would be interesting to know. Do they think that this sort of policy will even be effective? If so, then you could say, well, if the machine learning researchers think that a chip ban will in fact slow Chinese progress, at least that could be something that the executive could take into account. But on the other hand, if they think it's not even going to work, then it's like, well, jeez. Now we're just sitting here escalating, and the elite researchers, many of whom are, in fact, originally from China, don't even expect it to have an impact. I could see that being perhaps policy relevant at a super high level.

S0 (1:19:40) Yeah. I think if I was going to run a survey about that sort of thing, maybe I'd want to talk to chip experts rather than AI experts. Like, you you can just run a survey of any experts sort of to to answer your emails. Also, maybe I have some kind of hesitation around that sort of thing, or I'd have to think more about it. Partly just a lot of what AI Impacts does is on topics where we feel broadly just more people knowing what the situation is like in more detail will make things better. And I think once we get close to things that are adversarial, you have to be more careful. Like, you're probably helping 1 side or another. I don't know enough about the topic to have a a strong view on what should be happening, but don't want to be contributing to arms race, prima Farsi.

S2 (1:20:23) It's a tricky 1 for sure. I don't have a strong view as to whether it's going to work or not. My bias not knowing if it's going to work is that it seems too risky. Like, the current course is definitely an escalatory course. And if you worry that the the most likely flash point in the world is that China will blockade Taiwan or something, then preventing China from getting any of the chips from Taiwan certainly seems like it may make them more likely to do that. Right? Because what do they care if chip production is disrupted if they are getting them anyway? That's fairly crude analysis. Certainly could be disputed. But if you could inform and say, hey, by the way, the machine learning research community doesn't even think it's gonna work. But it seems like new information, given how escalatory the current politics are, probably is much more likely to serve, if anything, as a reason to deescalate just because we're already in this sort of cycle of escalation. So you could come with a finding that's like, yeah, your escalation is potentially going to achieve your policy objectives, and then people would stay the course. But I would say that it's probably more likely that you could have a deescalatory impact in the sort of gambling analysis of maybe I'll find something that supports escalation, but escalation already exists. Maybe I'll find something that supports relative deescalation. Maybe that is, in some ways, more positive expected value just because the current course is what it is.

S0 (1:21:47) Yeah. That's fair. But, yeah, also, I don't know enough about it to be confident that I should want deescalation. But, yeah, as a baseline, don't love escalation.

S2 (1:21:56) That's another big question. I do think that it it could be clarifying there too. I could be convinced if I was really sure it was gonna work, then I would be more likely to support it. You know what? The worst scenario is a highly escalatory move that doesn't even achieve its intended goals. So it's at least gonna achieve its goals, then, like, maybe some escalation could be worthwhile if you believe in those goals. But even if you believe in the goals, if it's not gonna work, then it's just, like, a total net negative. So, anyway, food for thought for possible additions to the survey next year. Anything else that you wanna make sure people don't forget about this work or any other kind of commentary you wanna bring us to a close with?

S0 (1:22:32) I'm amidst quite a lot of uncertainty here. I think the chance of things just staying pretty similar and nothing coming of this to, like, drastically change people's lives seems quite low. So even if you're not very directly involved with AI, that seems like a pretty important thing to be keeping an eye on and trying to have accurate opinions about. What the public thinks about these things will affect what policies happen, and they think that could change the rest of the future forever a lot for better or worse.

S2 (1:23:00) Does that suggest that you'll be exploring some public opinion investments as well?

S0 (1:23:05) I'm sure some other people are working on that. I think a public opinion being important is somewhat different to it being important for us to know what public opinion is. So I guess the public knowing what public opinion is is potentially good for moving toward whatever view it's moving toward, hopefully, more reasonable 1. If you start to get evidence for a thing, and at first, you're like, nobody else thinks that. And we'd be able to speak quiet about that. It's, you know, helpful to be polling what do people actually think along the way.

S2 (1:23:33) Yeah. This work is so in the weeds and works so hard on the definitions. Obviously, pulling the public, you would have to have a very different set of questions and a much higher level sort of gut check for many people. But I do think the common knowledge is is probably pretty valuable, and I I do think also for just reassuring policymakers that they're not out on a crazy limb if they wanna do something, that the public is kind of with them by default, I think is probably pretty good to establish. Also, among the tech people, there's, like, a, I think, underappreciation of how skeptical the public is. I'm not sure that the Silicon Valley set is really realistic about what, like, my mom's friends think about AI.

S0 (1:24:16) Skeptical in the sense that, like, they they don't like AI.

S2 (1:24:19) Yeah. There there's a lot of people out there that are just like my mom's 1 friend who I've known for my whole life. Her reaction to me was like, it creeps me out. I don't wanna have anything to do with it. Full stop. Like, curious, not looking to automate some tasks, not looking to get help on an email draft. I don't wanna have anything to do with it. You could say, like, hey. They might have said the same thing on Facebook and have it on there too. So I don't know that that's the final word.

S0 (1:24:45) It is creepier than Facebook.

S2 (1:24:47) Yeah. No doubt. I do think there's just a a lack of grappling with just how widespread that sentiment is. Anything else on your mind? This has been a great conversation. You've been very generous with your time. I really appreciate that. And love the work, so I definitely hope to see a 2024 edition as well.

S0 (1:25:03) Thank you for having me.

S1 (1:25:04) Katja Grace, founder of AI Impacts.

S2 (1:25:07) Thank you for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.

S3 (1:25:25) Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now. At turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries from AI with cognitive revolution to Econ 1 0 2 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors, like Moment of Zen and my show Upstream. We're looking for industry leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erik@turpentine.co. That's erik@turpentine.co.