Full Transcript

Transcript

L-squared (0:00) They're having to spread their bets fairly widely because there isn't a real standout front runner in China for LLMs in the way that you might see just 2 or 3 really standout players in The US. You don't see as much research happening in China on AI alignment. You don't see top labs here committing to responsible scaling policies in the same way that Anthropic and OpenAI have. There's a lot less thinking here about corporate governance mechanisms that might need to be in place to mitigate the worst risk from AI. But maybe this is because you don't have in China such a long history of labs that are explicitly focused on building AGI.

Nathan Labenz (0:53) Hello, and welcome to the Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz joined by my co host Eric Torrenberg. Hello and welcome back to the Cognitive Revolution. Today I'm excited to share another episode in our experimental series providing wide angle coverage of critical AI topics. This time, we're providing a bird's eye view of the Chinese AI ecosystem. My guest is a tech worker and AI watcher based in Beijing who goes by the pen name l squared. She prefers to remain anonymous, not because her takes are super spicy. On the contrary, I think you'll find that they're very well grounded and not at all sensationalist, but just out of an abundance of caution given her location and employment situation. In this wide ranging conversation, we cover the major AI players in China, the quality of Chinese models and products and how they compare to US offerings, the Chinese public's engagement with AI products, the state of AI regulation in China and the Chinese government's general stance toward AI technology, the impact of The US chip ban both today and in the years ahead, the state of US China AI research collaborations, and much more along the way as well. 1 of my key takeaways is that The US and Chinese AI ecosystems are in many ways more similar than they are different. In both countries, the major tech giants are leading the charge by writing the biggest checks to AI startups, providing critical infrastructure, and increasingly competing with those same startups in the race to bring AI products to consumers and businesses. Research in both countries is also being pushed forward by a mix of big tech, top universities, and startups. In terms of differences, the most important 1 is simply that there are no standout labs in China that are pushing the frontiers of possibility in the way that OpenAI, DeepMind, and Anthropic

Nathan Labenz (2:55) are doing here in The United States.

Nathan Labenz (2:58) Beyond that, l squared shares fascinating details on the current state of China's AI regulations, which on paper are quite onerous, requiring companies to jump through significant hoops to bring products to market, but which so far seem to be enforced only very loosely. We also discussed the apparent lack of jailbreaking culture that comes to the fore whenever a major new model or product

Nathan Labenz (3:18) is released here in the West.

Nathan Labenz (3:20) I am very grateful to l squared for taking the time to share her knowledge and perspective on all of this, and I also wanna thank Jordan Schneider of China Talk for making this connection possible. For more of l squared's writing on AI in China, you can refer to the China Talk newsletter. We did have some minor recording issues due to VPN bandwidth limitations while recording this episode, and so we are using an AI enhanced version of the web recording. I think it sounds very good overall. That's a testament to the power of modern AI systems. But if you hear any weirdness along the way, that's why. As always, if you find value in this work, please take a moment to share it with friends. Regular listeners will know that while my politics are generally very libertarian, and I certainly would not want to live under the Chinese government's restrictive speech codes, my bias is to look for ways to deescalate US China tension. And I hope that content like this, which provides an up to date on the ground perspective on the state of Chinese AI technology, can serve as a counterweight to the worst fears that our collective imaginations might conjure up. Finally, if you've got a unique perspective on another important area in AI, even if that's just a contrasting perspective on the Chinese AI ecosystem, I would love to work with you to make an episode about it. So please reach out via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now I hope you enjoy this insider's view on China's AI ecosystem with L squared.

Nathan Labenz (4:47) Knowledge worker and AI watcher based in Beijing, China, previously published on the China talk blog with some outstanding analysis. Thank you to Jordan for making this possible with a timely connection. Welcome to the cognitive revolution.

L-squared (5:02) Thanks so much, Nathan. It's great to be here.

Nathan Labenz (5:05) Yeah. I'm really excited for this conversation, a conversation from 12 time zones apart and across global spheres of influence with a focus on what is going on with AI in China. This is something that I've been trying to wrap my head around a little bit and just find it very difficult to get a qualitative understanding of despite trying to read the blogs and and whatever else is available to me. So really excited to get your take. I basically just wanna come at it from all angles, and that includes, like, your experience as a user of some of these products, watching society there more broadly, your take on what the government is trying to do and what they're actually doing, and a little bit of how you think the international competition is shaping up as you see it from Beijing. How's that sound?

L-squared (5:53) Sounds fantastic.

Nathan Labenz (5:54) Alright. Let's do it. So I guess, obviously, a big disclaimer is I really don't know what I'm talking about a lot more than I do, but this will still be obviously a limited perspective. 1 of my favorite I don't know if it's a joke or proverb about China is that man goes to China for 2 weeks, thinks he knows a lot about China, stays for a year, realizes he knows nothing about China. I'll put myself very firmly in the nothing category. But you wanna just give us a little bit of a a context on your experience in China, like the kinds of work you've done there, how long you've been there, that sort of thing?

L-squared (6:26) Yeah. Sure. So as you may tell from my accent, I did not grow up in China, but in London, I've now been working in Beijing for the past 5 years or so in a mixture of Chinese tech companies and Western ones. And I am now focused more on the infrastructure side, cloud computing, but I'm still following Chinese AI of Chinese language skills that have allowed me to follow what is going on here. But I really like that you shared that quote, and I do find myself that there's still so much more I need to know and, yeah, really recognize that I'm only sharing 1 limited perspective here today.

Nathan Labenz (7:13) Cool. If anyone's listening and, wants to offer a contrasting perspective, we could certainly welcome that, voice on the show as well. So, definitely, if that is you, don't be afraid to reach out. For now, let's start off with just an overview of the Chinese AI ecosystem, if you would. I think, obviously, in The US, people are familiar with your OpenAI's, your Google DeepMinds, your Anthropics as the sort of leaders, and everybody else is paying attention. But those guys have really driven the agenda. I'm very curious about how you would describe the situation correspondingly in China. Like, who are the live players, if that phrase makes sense? Who are the big companies? Just give us kind of that very basic background overview.

L-squared (7:55) Yeah. Sure. So first off, the caveat that AI is obviously a very broad set of technologies. And I think for most of our discussion, we'll be focusing more on the large language model providers. So within that base, I would break things down into 2 main groups, you've got the big tech companies and the LLM startup. So on the big tech side, you've got companies like Alibaba, which is a big ecommerce player and Tencent, which developed the WeChat super app. Both of these companies have their own large language models, but have also been investing in LLM startup. And Alibaba in particular has been very active here. It's invested in at least 5 different LLM startups. And it's important to note that they are also the cloud computing provider in China with the biggest market share. And so we see with these investments that they are likely getting the startups they're investing in to use their cloud resources. And they're having to spread their bets fairly widely because there isn't a real standout front runner in China for LLM in the way that you might see just 2 or 3 really players in The USA. As well as Alibaba and Tencent, you've got Baidu who are the search giant in China. They have their own Ernie chatbots which is doing pretty well. We've seen that at the top of the Superclue benchmark which is a prominent benchmark for Chinese language AI models, and ByteDance, who people may know as the developer of TikTok and donor Chinese equivalent, they also have a bot product that seems to be doing pretty well according to 1 source I saw recently that was at the top of the monthly active user leaderboard for Gen AI apps in China earlier this year. And then moving on to the LLM startup, you've got maybe 2 categories that you could further break things down into. So 1 is startups that have emerged from Tsinghua University, which is really China's top university for computer science. And there's a startup called Moonshot AI and another 1 called Zhipu AI that are both started by Qinghua professors and have managed to attract quite a lot of funding and attention. And they're fairly small in terms of employee size, maybe couple of 100 people really able to produce pretty decent models. And then you also have some startups that are started by more established entrepreneurs. So people may have heard of the team that was started by Kai Fu Lee, who is a pretty prominent tech leader in China, has experience at Google China and Microsoft Research Asia. And their set of models, the e models have been doing pretty well on open source leaderboards. And you also have startup called ByteDance, which was set up by the founder of a search company in China. And, they've been investing in both open source and closed source models.

Nathan Labenz (11:34) So we're just summarize or attempt to compare and contrast that to The US situation. Overall, it sounds more similar than different. I think you named 4 big tech companies that presumably have a lot of cash on the balance sheet and can pretty freely write these investment checks. And they are also the owner operators of the physical capital that all this stuff has to get trained on and eventually run it at inference time on. And it sounds like they are all wise to the opportunity and very much engaged with it. And then at the startup player, as you said, maybe the big difference is that there isn't such a clear inner inner group of leaders, but there are sounds like a, you know, pretty healthy number of startups, including from top universities and just known people. So is there anything else any anything I'm missing there, or would you say overall, aside from that kind of nobody's like the clear front runner caveat, would you say it is essentially a similar business dynamic?

L-squared (12:42) Yeah. I'd say there are similarities. I should note that I haven't given a comprehensive list there. There's other big players you could mention like Huawei, who are especially important on the infrastructure side, also have a big cloud business. And there are other startups as well. And this has led to some people in the Chinese ecosystem commenting that there are too many players in the foundation model space, and this is a waste, especially given that China is facing quite severe constraints on the hardware side. I think it is very possible that you'll see narrowing down of players in the coming months and years, more concentration happening over time.

Nathan Labenz (13:26) Yeah. Certainly. A lot of companies have gotten started here in The US as well, and I don't think all of them will ultimately make it. So I think that definitely also sounds pretty familiar. We'll circle back to the hardware question in a little bit. I wanna definitely ask some questions about how much the the chip ban or just chip restrictions in general are already having an effect and how that might change over time. But before that, how would you say the public is engaged with AI in China? Here, I'm very much in a bubble, obviously, so it's all I talk about and all I think about and everyone that I know is, like, trying to maximize the performance of language models. Obviously, there are still lots of people in The US that have not even used ChatGPT even once or a lot of people that used it once and didn't didn't get hooked immediately. But it is definitely something that has cracked the even if not everybody's using it, it is something that has really cracked the public consciousness. Like, it's something that made the state of the union this year and is all over the news, and there's a White House executive order. How would you describe the kind of just public awareness in China? Has there been the sort of ChatGPT moment, if you will, in China?

L-squared (14:37) Yeah. Like you, I am in a bit of a bubble by itself, but I do think there is broader public consciousness about the rise of ChatGPT and and similar tools in China, discussions of AI on the mainstream news. There's been discussion both of ChatGPT and more recently Sora and the need for Chinese equivalents to be developed. And I think people do feel that this is really important technology both at the government level, but also the wider public paying attention.

Nathan Labenz (15:16) So are people like is it is it on TV? Is it the sort of thing that is it's just on everybody's minds? And would you say there is a, like, a culture of people using it at work throughout the economy? Is there, like, a leader? Is there a brand name like a ChatGPT that people would most most readily know or most readily go to if they were gonna seek out a language model solution?

L-squared (15:39) Yeah. In terms of whether it's on TV, yes. Definitely see segments on the news about this. Recently, China had its biggest political meetings of the year. And around that time, you had political leaders going to visit places like Baidu and the Beijing Academy of AI. And I see reports of those visits on the news. And we've also seen us AI generated cartoon series that was broadcast on Chinese TV. So lots of companies, including state owned enterprises, looking to make use of AI GC services. And the other part of your question was whether people are using these tools at work. And if there's a go to provider, I wouldn't say that there is a clear go to provider for businesses here. I think that, as we discussed with the last question, it's still fairly open landscape here. And I think there are a number of different companies that are selling to businesses doing proofs of concepts, and it's still a bit early to tell who those leading providers might be. You want a sense of how widely AI assistant tools, how widely used they are. I mentioned that ByteDance's chatbot had been at the top of the monthly active user leaderboard earlier this year. I think that had about 17,000,000 MAU. By comparison, there's estimates of ChatGPT having about 27,000,000 in The US. So, yeah, maybe there's not quite as much take up yet in China of these AI assistants, but it's comparable.

Nathan Labenz (17:31) Yeah. That's interesting.

Nathan Labenz (17:33) Hey. We'll continue our interview in a

Nathan Labenz (17:35) moment after a word from our sponsors. Obviously, the population size of China compared to The US is something like 4 x. And I wonder what you would consider to be the addressable market multiple. I don't have a great command of Chinese demographics, but I would if you segmented it to, like, how many people are is it like a tier 1 city? Would that be the sort of way to think about it? Or tier 1 and tier 2? There's I know that, again, I'm showing my ignorance of China here, but I know that there are these sort of systems where not everybody can move to Shanghai. Right? So there's different classes of sort of economic privilege within the country. If you were to say, what is the sort of universe of people who are, like, living broadly a sort of first world life style or first world work style that would be in a position to use a language model? Is that, like, half of China?

L-squared (18:28) Good question. Yeah, I think maybe around 40% of the population would be kind of China's middle class 4050% might say. So 600,000,000.

Nathan Labenz (18:46) Gotcha. Okay. So I guess the takeaway there is it's pretty early in the adoption curve still in both countries. 27000000 in The US is still under 20% of the working the workforce in The United States. Obviously, not everybody's in a job that would benefit from a language model, so maybe that's, like, getting up toward 40% perhaps of people that, like, could conceivably use it, which I guess is not not nothing, especially given how little time it's been in the market. It sounds like maybe there's a 2, maybe a 3 x kind of multiple in terms of per capita how many people are using something like this in The US versus China, but not like a totally night and day sort of difference. How about the tools themselves and just the quality? In preparing for this, you were kind enough to take me on a little journey into the world of Baidu's Ernie. You can maybe give us the Chinese name for that, but I know it as Ernie 4. And this was a couple months ago already. So we were looking at leaderboards and trying to get a sense for how good is this thing and is it close to GPT 4. They, I believe, Baidu themselves put out a benchmark called super clue, which is meant to be a more Chinese language focused benchmark, but still they compare themselves to GPT 4 on that benchmark and find that GPT 4 is still tops in the even in the Chinese first benchmark that they are measuring themselves by. But at the time, Ernie 4 was the best of the Chinese products on that benchmark. And I guess I would we didn't do a ton of testing, so you can expand on this. But in the little bit of testing that we did, I probably was pretty impressed and and came away feeling like this is a pretty comparable to ChatGPT sort of product. We ran a test prompt on a coding exercise, and the quality of the Python that came back from ChatGPT was definitely not head and shoulders above what we got from Ernie 4. Again, it's little hard for me to evaluate because I don't speak Chinese and the code is code, but the the comments are in Chinese and all of these little nuances to it. But I guess my first order summary based on that experience was they seem more similar than different. What more can you tell us about the experience of being a retail user of the Chinese products?

L-squared (21:11) Yeah. I think that's a pretty good assessment. I would just clarify quickly that the Super Clue leaderboard is not something that is developed and managed by Baidu itself. It's a third party that is doing that leaderboard and updating the results every couple of months. But you're right that Ernie, Baidu's product is at the top of that leaderboard in terms of the Chinese models. And, yeah, my own experience playing around with some of these models is that they are pretty good, pretty useful thing, for Chinese language tasks. The gap with GPT 4 is not huge. I'd say there's some differences in usability among the Chinese apps that I've been trying. So some of them are overly sensitive in terms of not responding to questions that I think seem quite harmless, not political at all. But maybe I've accidentally triggered some keyword filter that they have in place. And then I just don't get the answer that I'm needing. So that is 1 limitation that you get when using these tools in a censored environment. Also, some of the tools don't have context windows that long. I think there is a bit of a usability gap in that sense as well.

Nathan Labenz (22:43) How about availability? Is there ever and this is 1 way to get perhaps at the tip question. Is there a is there contention for resources? If you log in, do you ever have it say, oh, hey. It's overloaded right now. You can't use it, or you come back later? We ChatGPT has the occasional outage, but probably speaking, it's, like, available, and it doesn't seem that they are as much as they may say they're constrained on chips for a trading GPT 5, it was like there's plenty of ChatGPT to go around is my first order approximation. Does it feel the same way in China? Have you subscribed? Do you just have consistent access, or is there anything that seems like it would be an echo of the chip restrictions?

L-squared (23:26) Yeah. I haven't faced availability issues when I've been trying to use these tools.

Nathan Labenz (23:31) Do you know anything about tokens? That was also a curveball question I didn't prepare you for. But in the the language models that I'm most familiar with, a typical vocabulary size would be like 50,000 tokens, and they're all English words and word parts and so on and so forth. Some very strange ones as you get into the very deep long tail. But I think Chinese characters are basically all multi token in an English first vocabulary context. Do you know if any of the Chinese providers are doing their own tokenization, like, creating their own vocabulary and perhaps, like, predicting Chinese characters in a more token efficient way as a result?

L-squared (24:13) I'm afraid it's not something I'm familiar with.

Nathan Labenz (24:16) I'll have to I'll have to check on that 1 myself. That may be something I can answer. So put a bookmark on that 1. Sorry. I also just checked the leaderboard lmsys.org, which is my as regular listeners will know, my go to for comparing new language models. I I do find that to be broadly very trustworthy. Cloud 3 has jumped right to the top right up alongside the latest GPT 4 turbos. Right now, the number 1 Chinese model on that leaderboard is Qwen 1.572 b, which I believe comes out of Alibaba. Is that something that you would go to an Alibaba chatbot to access? It's the license said there that it was, like, a special license. So it seemed like it was not open source or maybe it is open source, but under an a custom license, which I know the these companies sometimes do. Did that model, like, make waves when it dropped? And how would you access that if you wanted to go use that model?

L-squared (25:15) Yeah. So it is marketed as open source, but I haven't looked at the details of the license agreement. So, yeah, it's very possible that there's certain constraints around it. That mean, there's not really that open source. In terms of how the average user in China would access that, Alibaba has its own chatbot product called Tong Yi Qianwen. And this is probably similar to Qwen, but it's, I guess, more advanced closed source version. So you wouldn't really as a typical user go and use that Qwen version that we're seeing on that leaderboard. I guess that's more for the developer community who could access it through platforms like Hugging Face. But, yeah, for the Chinese users on the consumer side, they would be accessing this slightly different model, which I guess has more moderation mechanisms in place to make sure it is compliant with Chinese AI regulations.

Nathan Labenz (26:20) What does the price look like on these subscriptions? I think when we looked at the Ernie product, it was, like, equivalent of 8 US dollars a month. Is there it seems like in The US, it's like everything is $20 a month at the retail level. That's true for ChatGPT Pro. It's true for Claude Pro. Replit Ghostwriter is there now. You.com is there now. I think Perplexity is at the same price point. Seems like everything is right at that $20 a month. Has there been a similar sort of convergence on this on a single or very common price point in the Chinese market? And if so, what is it? I guess it would be the $8 if that's what it is. But has there been that convergence?

L-squared (27:02) Yeah. I'm not sure. I haven't done a comprehensive check of the price point of these different models. My impression is that a lot of them are still offering services to consumers for free, probably still at that stage of wanting to grow their user base. So Baidu is the main 1 that I know of that is around that $78 mark and has been commercializing for several months now. But I think might be a little while before other services are charging consumers at that kind of level. Some people in the investor community here are saying that really the enterprise facing AI apps are aware the money is gonna be made, it's gonna be pretty hard to commercialize these consumer facing apps.

Nathan Labenz (27:59) Interesting. Yeah. I think certainly there's gonna be a lot of inter enterprise revenue in The US as well. I guess I do think a lot of people ultimately are gonna get these tools from their employers. So in The US, maybe people are a little more willing to subscribe to some newfangled thing. But in the end, I do think most people probably get it from their job just like they get Google productivity suite or Microsoft productivity suite from their job. This seems like probably something that ultimately is part of a package of tools that you get. So that seems maybe, like, we're headed for similar places even if it's little bit of to get there. I the Qwen 1.572 b, I I should emphasize too, is according to the LMSYS leaderboard, really neck and neck with the original GPT 4 generation. So right now, you've got GPT 4 turbo and cloud 3 as the clear top. And then you drop down to that next tier, and it's GPT 4. And and Qwen is right there. It's in I think it's in the top 10. It's just slightly before the earlier generations of GPT 4. So that really is quite good. It's also remarkable that's open source. If it is open source, it might be the global leader in open source. And it's also remarkable that it is, I would assume, predominantly being tested in English on the LLM SYS arena because that is an open place where people can just come and run whatever prompts they want. They on the back end will put it up put 2 different models head to head. You see the output from each, and it's your, you know, decision to choose which 1 you like better, and that's how they determine their rankings. I can't imagine there are too many people in China that are bothering to come to the LLM leaderboard, although I could well, it's it's a big world and full of surprises, but I'm assuming that this rating reflects largely English usage, which is suggestive of the possibility that in Chinese, it might even be a bit better. And it would seem plausible that new 1, again, this is the Qwen 1.572 b, that it might be a better experience for Chinese users than the earlier GPT fours. So it does seem like the frontier in each country is, like, not that different from 1 another. Or anything else to add there? But, yeah, it's an interesting bottom line that, you know, people talk about The US being, like, years ahead. That's a common notion. And we're gonna keep our lead with the chip ban or whatever. And I'm like, from what I saw of Ernie 4 and from what I'm seeing on LMS's leaderboard, there is not a years ahead difference. Now 1 caveat too is they're they they might be training on OpenAI outputs, which is definitely a fast path to refining your behavior. That's been a very common practice all over the world. And I don't know. I know there's been, like, a little bit of noise as to that might be happening among some Chinese providers. I think 1 even maybe acknowledged it to a certain degree. Have you heard any kind of noise or what would be your sense of to what degree Chinese companies might be drafting on OpenAI's work, so to speak?

L-squared (31:16) Yeah. You may be referring to a story about how ByteDance's OpenAI account was blocked because it was found that they were using OpenAI outputs for training their own models, which is against the terms of the usage agreement. I'm not sure how widespread this practice is more broadly among Chinese developers, but there is a lot of use of the LLM architecture. And recently, there's been a presentation that the Beijing Academy of AI delivered to the Chinese premier. And this presentation that's been circulated has set out 3 main challenges for Chinese AI, and 1 of those is reliance on foreign open source architecture. And so I think people here in China see that as a vulnerability. If we're to see US government attempts to impose export controls on open source technology,

Nathan Labenz (32:26) they're gonna have a hard time taking LLM back or preventing Chinese researchers or companies or whatever from accessing it if this stuff continues to be open source. So I'm not sure I quite get the fear there. I guess the the fear would be maybe it gets cut off. Right? But aside from that, it seems hard to that's 1 of the big I don't go in for this argument personally, but among the folks that want to see open source models curtailed, 1 of the arguments is that China is gonna be they'll they'll use them. Right? And so if we want to create this separation in terms of capabilities, then open sourcing it negates that. Am I missing something there in terms of why Chinese companies would feel like that's a risk?

L-squared (33:10) Yeah. I suppose, right, the cat's been let out the bag, and you can't really stop Chinese companies from using LLM and now. But I think the fear is more about if we look to the future and we see other important innovations coming out of the open source community in Western countries, if you have controls on that community and their ability to provide open source models to people in China, then, yeah, that can be something that limits China's ability to stay following the cutting edge.

Nathan Labenz (33:51) Yeah. I I could if we actually shut down open source, I could see that contributing to a widening gap at the moment. It seems like that's not close to happening. And, until it does, it seems like the and maybe even beyond that. I'm not saying I don't have a opinion on whether Chinese companies would in fact fall behind if the open source were to be cut off. I honestly don't think so, but it's interesting that you have somebody in the that somebody who knows more than I, do certainly seems to be concerned about it. So it's an interesting data point.

Nathan Labenz (34:21) Hey. We'll continue our interview in a

Nathan Labenz (34:23) moment after a word from our sponsors. So you mentioned presenting this stuff to political leaders. I have been fascinated by, first of all, just the fact that these models are getting open sourced. If I were to guess which country would be first to clamp down on the open sourcing of language models, it would seem like they would be China would be the first to be concerned about that just because once they're out in the open, the we've covered this in other episodes, but, like, they are not going to follow censorship rules once they're out in the open source world. Even if they have been trained to refuse to talk about certain things, any amount of fine tuning is typically enough to remove that. We've got an episode with Adam Gleave from far AI that talks about inadvertent jailbreaking where fine tuning a language model even without the intent to jailbreak it, but just to do whatever you're gonna do often removes these refusal behaviors and essentially resets the the censorship layer on the model. So how did are you surprised that there are so many open source models coming out of the Chinese companies with that in mind, or maybe that's just something that the government hasn't quite figured out yet? Or, like, how would you describe the maybe I shouldn't be surprised. I get it. I'm I'm just very confused as to if you'd asked me to guess, I would have guessed that there would not be a lot of open source models coming out of Chinese companies. But in fact, there seems like there's a lot. So maybe my assumptions are flawed or maybe the government hasn't caught up. What do you how do you just how do you account for that?

L-squared (36:02) Yeah. I think this reflect a difficult balance that the Chinese government is trying to strike, like many governments worldwide between really harnessing the benefits from AI and leveraging it to improve economic productivity, while also managing risks that it presents. And I think that it does see open source as important for helping to advance AI development, help China to catch up with The US in terms of its AI landscape. And so it's not cracking down on open source. But meanwhile, it is bringing forward AI regulations that it hopes can mitigate the worst risks from AI, which I think in its view, really these risks of destabilizing the country politically. So it's got pretty tight controls that service providers need to comply with in terms of making sure that outputs from these models are gonna be in line with the CCP's narratives and and not saying things that are politically unacceptable. This kind of risk is is very much on its radar and and something that it's trying to control by making sure that any models that have large user bases that are going straight to Chinese public are having to follow these quite onerous regulations, but it's just the trade off that the government is having to grapple with.

Nathan Labenz (37:40) Okay. You mentioned there the onerous regulations. This is something that, again, I feel very confused about. I've heard a series of different reports where I feel like last summer, there was a draft regulation that basically said you're gonna have to prove that your training data is all up to standards, which, of course, people were like, we're talking about trillions of tokens here. It's gonna be really hard to go through and verify that is actually true. How are gonna do that? Obviously, 1 way, interestingly, would be to use language models to do the filtering, but you don't have the language models. You don't have the compute to process the tokens. Like, it does become a pretty significant burden. Then later, there was an update as I understand it, and it seemed for a minute that had been relaxed. But then I read 1 of your posts on the China talk blog, and it seems like your take is that, no. In fact, it really wasn't all that relaxed. So help me again just get unconfused. Like, what are the regulations that folks have to go from? Do they actually have to prove the ideological purity of their training data at this point, or what are the kind of biggest hoops that folks have to jump through?

L-squared (38:54) Yeah, it's a good question. Because there has been some changing around in terms of what these regulations are actually saying. So the main upshot is that LLM providers or Gen AI service providers would need to do a registration process, which will involve a series of safety related tests on training data and on model outputs. And only once they have shown to regulators that they meet certain thresholds of acceptability, can they get the approval that they need to deploy their models? In terms of the training data, it's not the case that they would need to go through and sample every single piece of data, there'll be requirements around sampling a certain subset of their training data, and not using batches that are failing these safety tests. But yeah, it's still quite onerous, I'd say. What is worth noting is that you do have a distinction between what these rules say on paper and what is actually being enforced. 1 example that relates to training data is that recently, we've seen a security standard come out for generative AI, which is really putting a lot of detail on how companies should actually be complying with these Gen AI regs that came out last year. And in this security standards, it says that providers should not be using training data from sites or sources that are blocked according to the laws and policies of China. Anything that is blocked by the firewall in in theory, should not be used for training data and currently hugging faces blocked in China. So is Wikipedia. And if this were really enforced, then that's quite prominent sources common sources of training data that these Chinese companies are just not meant to use. And so I think it's quite unlikely that this is gonna be something that the government checks up on. I think it's more likely that developers will be continuing to use these sources, but just not telling the government that. And I think the government will probably turn a blind eye because it doesn't want Chinese Gen AI services to be falling too far behind. Another thing I would know is that so far, you've only seen about 40 apps and services that have been approved under these Gen AI regulations. And I think that there are probably many more Gen AI apps that are actually in use that are being served to customers and businesses. And this suggests to me that there are a lot of companies out there that are taking a look at these regulations and the kinds of hoops that need to be jumped through to get approval. And they're just deciding that it's too much effort, they don't have the bandwidth to go through this process, which can often involve a lot of back and forth with the government. And you also see some companies saying that their tools are just open for internal testing, you have to apply for access, get on the whitelist before you can use them. And I think that's a kind of risk mitigation measure where if the government comes to them and says, haven't registered, what are you doing? They can say, oh, we're only opening this to a limited number of users. And that would maybe help to reassure the authorities. But in practice, they're approving anyone who applies for access. And it's really not that much different in practice from full deployment. I think that, in summary, you do have pretty demanding rules for Gen AI services in China. And this can be a barrier for companies that are looking to quickly grow their user base. But there's also a gap between the law and and what is being enforced on the ground. And so there there is still some space for companies that are starting up in this area.

Nathan Labenz (43:32) Yeah. That's, again, just very confusing. And the idea that the certainly familiar reality that the law says 1 thing, but you can get away with something else in many cases. I would say that's true here in The United States even just as I, roll through a stop sign in my neighborhood from time to time. But here, it is pretty striking. We went to this I went to this 1 Yee chatbot that was I think it's now I think it's still the second rated 1 on the LMSS leaderboard. This is the 34 b, and this is from the company that was started by the guy you mentioned earlier, Kai Fu Lee. Interesting background. I'm just reading into his story a little bit. I understand he's Taiwanese, but the company's headquartered in Beijing, which is interesting right off the bat. The model that they put out, they said was trained on 3,000,000,000,000 tokens. Notably, that's 1000000000000 more than LLM 2. They're definitely going beyond Lama 2 with that. But when I went to it in the LMSS Arena and just chatted with it, I, of course, asked it about Tiananmen Square, and it just gave me the answer that I would expect. It was like that I would expect from western chatbot. It should to give you slight taste of it. On 06/03/1989, the government decided to take decisive action to end the protests. The military was deployed into Beijing, and on the night of June 3 and into the morning of June 4, soldiers and tanks entered Tiananmen Square, forcibly clearing the area. The exact number of casualty casualties is unknown, but estimates range from several 100 to several thousand people killed. It goes on, but, obviously, that's pretty sensitive content in the Chinese context. Would you would explain that by basically just saying that, hey. The rules are 1 thing. They're getting data where they can get it, and they're just training without doing those sorts of filters. And that's the that's the natural result if you just train on the open Internet. Is that your theory of that case?

L-squared (45:32) My theory would be that for this stage where Chinese developers are putting open source models out on platforms like Hugging Face, they are not doing as much to ensure that they are in line with Chinese standards and rules. So maybe not putting the same keyword filters on that they would put on there if they are serving Chinese customer base. And so recently, you've seen that company put out a API that Chinese companies can use to access the e 34 b model and other models that they put out. And I would bet that with that API, there are more measures in place to make sure that you wouldn't be able to get that kind of answer on Tiananmen Square. So I think there is just a case of hugging face not actually being publicly accessible in China. There's not that need for a company to comply with those same strict content restrictions that they would when they're more clearly serving the Chinese market.

Nathan Labenz (46:48) Yeah. Interesting. Okay. There was a good post on China talk about this recently as well. And, basically, what they did in this post was a rundown of a handful of top models and just asked it a bunch of asked each 1 number of sensitive questions and try to systematically get a handle on how the different Chinese language models behave. To summarize that, it was notable, first of all, that there is quite a range of behavior. I I think the Yi 1 that was the same 1 that gave me the Tiananmen Square output was the most western in its orientation in the in just the nature of its answers. But there were definitely others that showed a much more kind of what you'd expect from a a Chinese product. It seemed they also reported that there is a difference to your point between what you

Nathan Labenz (47:43) on Hugging Face if you're

Nathan Labenz (47:44) just looking at the bare model and what you would experience in the productized context as it exists on the Chinese sites. The explanation for that seems to be that there is an expectation for app developers that they're going to do some filtering alongside the core generation. And this is something that we've seen leaders in The US do this to a degree. Like, the original Bing had a feature people may remember where it would generate an answer. And then at some point, if it detected that it had gone off the rails, it would retract that answer, you would actually see your answer disappear from Bing, and it would be replaced with a sort of, sorry. I can't talk about this right now thing. So that seems to be honestly pretty rare in The US or in the in the West, let's say. Most apps I go to do not do that. And even, you know, the ChatGPT's, the Google Gemini's, like, they pretty much just generate whatever they're gonna generate and and give it to you. They don't really seem to do much in the way of filtering. It seems like that's actually quite expected in the Chinese context, and they see the difference between what you experience on a a hugging face versus, again, the the Chinese retail apps. They also reported that the performance did not seem to obviously correlate with the behavior in the sense that my takeaway from this was RLCCPF can work, meaning reinforcement learning from Chinese Communist Party feedback. Like, it it did not seem that the models that conformed more to the expected standards behaved worse. In the reporting on the China talk report, it was basically that the performance didn't seem to be affected by those control measures. So I thought that was quite interesting. And, of course, these are language models, so there's still, like, plenty of weirdness. It was only a certain sample size, and you get models that sometimes answer 1 way and sometimes answer the other way. So it's like there's definitely plenty of weird stuff still to figure out. It also would vary if they were prompted in English versus Chinese. So definitely a lot of just a lot of weird stuff to be explored. But I do recommend that post on the China talk blog if you wanna get a rundown of the leaders and some of their behavior on a small but structured task. I'm sure you read that. Did you have any other thoughts or takeaways from that that post on China talk?

L-squared (50:13) Yeah. 1 thing to add, you mentioned that he was the model that seemed the most Western in its answers. It's important to note that when that test was done, they didn't have a consumer facing platform in China. The only version that was tested was that hugging face version. And as I was saying, I think you would see something different once you test the China facing interface. Think that you were summarizing there how you didn't see a huge performance hit from this kind of censorship. I think that times roughly with the take I gave earlier about how I'm still seeing value from these tools, even if sometimes they do set off a sensitive keyword trigger and you can't quite get the answer that you wanted. But yeah, I think there is ultimately this tension between trying to be useful and not touching the red lines that the government has put in place. And I mentioned earlier that recently we've seen a slide from the Beijing Academy of AI circulate which summarizes its view of the main challenges from AI in China. 1 of those I said was the fact that the underlying architecture of many of these models is not Chinese. Another 1 they mentioned was the lack of controllability of AI models. And so I think this is something that the Chinese developers are still struggling with and this can be pretty high stakes seen Chinese government take quite severe measures against entrepreneurs and tech companies in recent years. So I think there is always that risk that you have some high profile story come out of how a Chinese chat bot said something that was rude about a Chinese political leader or in some other way unacceptable. And that could be quite damaging from a PR perspective or even lead to temporary or more permanent shutdowns by the government. I think that kind of risk that developers face is much higher compared to in The US.

Nathan Labenz (52:30) Yeah. How is there a difference between when you said earlier that there were, like, 40 providers, is that a foundation model provider, or would that be, like, an app? So for example, I'm involved with this company, Waymark, which I started. Blah blah blah. People have heard me talk about it. We, of course, are an app on top of the foundation models. We are fine tuning them, but we're not we're not developing the core technology. We're productizing it. We would we be subject to those same sorts of preapproval requirements there, or is there also an app layer that once a language model, foundation model provider is approved, then, like, other people can use their stuff via API in random consumer apps? It's not clear to me, like, if there's a 2 tier system or if everybody has to jump through all those hoops.

L-squared (53:23) So the app developers are also meant to jump through those hoops. I am not quite sure how much detail they are expected to go into. It's possible that if they are building on a foundation model that has already been approved, that it might simplify the process a little bit. But, technically, they are still meant to go through safety assessment and and registration. But as I say, it's likely that because there are so many apps and only 40 registration so far, it's probably a lot of companies that are not actually choosing to go through that process even though they should.

Nathan Labenz (54:05) And so how do they distribute their products? I guess I've again, I'm showing my ignorance of the Chinese system, but you could just put up a a website and people can come visit your product, and not every website has to get checked. And then, like, on a mobile app here, if you're gonna publish a mobile app, you gotta go through Apple review or the Android store. And in each case, there's some, in theory, some sort of a check there. Right? So is there I imagine there must be a comparable thing if you're gonna actually have a native smartphone app. So would these folks that are that you're this sort of dark dark matter of AI apps, are they likely just to be found on random websites that just nobody's really looking at?

L-squared (54:44) Yeah. I think that they would still be reaching customers through Chinese app stores, through their own websites. And although these app stores should probably be checking for this approval, maybe in those kinds of checks are not really being conducted. But yes, possible that that might be something that is cracked down on later on the government decides to do a cleanup of the apps. But maybe 1 factor is that this process for approving apps is still pretty new. We've seen reports of taking several weeks, months even for the developers and regulators to exchange information to make sure that the regulators are happy. And so there's probably a big backlog of applications. And, yeah, maybe the government recognises that right now, it's not super realistic, these apps to be registering fast. So they're not being that heavy handed with the enforcement.

Nathan Labenz (55:57) Is there a culture when fun ritual we have here in The United States and, I guess, in the West more broadly is, although it feels like a very sort of Bay Area phenomenon in some ways, is the ritual jailbreaking of new systems when they come out. And we had this on the original ChatGPT launch day. People were finding all sorts of loopholes reminded of pretending is all you need. When it was like, oh, write me a bedtime story about hotwiring your car or whatever. And more recently, we've had the Gemini making black George Washington's and stuff like this. And everybody gets a good laugh out of it. Right? It's, I don't think it actually matters nearly as much as the commentary it broadly does aside from as an indication of the fact that the companies don't have a working way to control their systems. I don't I think that's not to be glossed over, but I actually don't think it ends up mattering all that much for their business prospects. But is there anything like that in the Chinese context? Are people, like, going and trying to get likes by showing what they managed to get Ernie 4 to do or anything like that?

L-squared (57:06) Yeah. I haven't seen this. It does seem to be quite different from the Western ecosystem in that sense. 1 possible reason is just that there's more caution here from Chinese netizens about what can and can't be said online. And you wouldn't want to go and share screenshots of your attempt to make a chatbot say something that's politically unacceptable. And yeah, maybe there's also some awareness that these Chinese models aren't quite that level of a GPT 4 or Claude 3, and it's obvious that they're gonna make mistakes. And maybe people just don't see as much surprise factor getting these models to do something that they shouldn't.

Nathan Labenz (57:59) That's interesting. How permeable is the firewall in general? If I am in China and I want to use ChatGPT, all the services are pretty much all the western services are blocked in China. Right? But is it easy to get around that, or is that now actually pretty difficult?

L-squared (58:17) Lots of people do have a VPN that will allow them to access foreign websites and services. I think maybe the main challenge with accessing, it's like ChatGPT might be the need for a foreign SIM card, but I think there are ways around that. I think you can buy the use of a a foreign number on e commerce sites in China, like, Taobao. So, yeah, there are friends that I know who are using ChatGPT. And, I think that for enterprises in China, there may be ways to access foreign LLM services. If you are a customer of Microsoft Azure, then I believe that they give access to the OpenAI suite of product to users anywhere in the world, including China.

Nathan Labenz (59:18) Interesting. That phone number bit, is that just to verify your account?

L-squared (59:23) Yeah. I think so.

Nathan Labenz (59:25) Yeah. Okay. Yeah. It's interesting. I tried to sign up. It was mostly a total fail, but I tried to sign up for a few Chinese products from here in The US. I was able to access the websites. I was able to understand what was going on with the help of Chrome's native translation features that didn't always clarify what was going on, but it was enough to allow me to try to proceed through a sign up process. But then I got totally blocked on the need for a Chinese phone number to receive an SMS to be able to confirm. And so I just couldn't go any further. Is there would I be able to do something similar in the reverse? Could I buy a Chinese SIM card or have some sort of Chinese phone service receive a text message for me so I could go try some of these things in more depth?

L-squared (1:00:13) Yeah. Good question. I would imagine that it is possible somehow, but I need to investigate that myself. Not sure.

Nathan Labenz (1:00:22) Yeah. Okay. We'll put that on my to do list. Okay. So I guess if I had to summarize everything so far, I would say it's more similar than it is different. The big tech companies are the infrastructure owners. They are the check writers. They are the brands that do a lot of the retail and enterprise distribution of these products. They also do R and D. They they partner with startups that are also trying to get to the cutting edge of capability. And people are definitely well aware of what's going on and more and more people are using them. But it also at the same time has not gotten to every last user or reaching, you know, any anything approaching saturation. Capabilities are broadly similar in terms of what's available to an end user even if perhaps the Chinese ecosystem is somewhat dependent on open sourced R and D from the West to get there as fast as they have. Point remains that they pretty much have, it seems, and the the a Chinese user is not, like, super far behind a GPT 4 in terms of the quality of support that they can get from a language model. There is this, like, approval process that is onerous if followed to the letter, but it sounds like it's not usually or often not followed to the letter. And if anything, maybe the biggest difference was the lack of the jailbreaking ritual and the sort of shit posting online. Aside from that, it seems you can call it maybe a bit of a warped mirror, but it it sounds seems like the 2 ecosystems are, like, closer to mirror images of each other than they are to very different very different realities. Would you agree with that summary and, you know, would you complicate it at all?

L-squared (1:02:11) Yeah. I think that's a really good summary. What might complicate things is the impact of the export controls going forward. So far, I'd say that we haven't seen it at a huge amount in terms of the fact that we do have pretty capable models still in China. And that's down, I'd say, to a couple of saving races for Chinese AI, which is 1 that many Chinese companies did stockpile a whole bunch of chips before the export controls took effect. So you've got cloud computing providers here in China that are still able to write access to good NVIDIA chips. And second, Chinese companies are currently able to use cloud computing services overseas. So they can train models using top chips that would be unable to be imported now into China. But going forward, those stockpiles are going to run out at some point. And meanwhile, the cutting edge of development is still moving ahead outside of China. And so you would expect the full effect of the export controls to become more visible after 2, 3 years and people who say now, oh, the chip bands clearly haven't worked. The Chinese AI is still good. They need to be a bit more patient. And then on the cloud computing side, you also do see the US government noticing that kind of loophole and thinking about how to cut off access for Chinese companies to the use of cloud services in The US. And so there was a a draft rule put out by the US government that's currently out for comment that would require cloud providers in The US to keep a record of when foreign customers are using their services for large training runs. And there's a provision in those rules that could allow use of these US cloud providers to be restricted for for people from certain countries, which would probably mean China. Yeah, I think that there is that real risk for Chinese developers that access to compute is going to get even trickier. And so that would be 1 thing to keep an eye on that could could cause that gap between Chinese and Western AI ecosystems to grow further in future.

Nathan Labenz (1:04:57) Yeah. That's interesting. We're just recording on Monday, March 18, my time. I guess it's Tuesday morning, March nineteenth for you. The Blackwell architecture was just announced by NVIDIA today with quotes from, I think, the CEO of every major big tech and and AI company. Hey. I haven't observed the details of that yet, but it does seem to be another meaningful advance in terms of the just raw volume of compute that it packs and also the energy and overall cost effectiveness is supposed to be an order of magnitude improvement as well. So I have been among those that maybe haven't been patient enough in terms of my interpretation of the the impact of the chip ban. Not which is not to say that I'm, like, wanting this to happen. I'm not I'm not sold on the idea that anyone is really gonna win by slowing down the Chinese ecosystem, but I have been probably in terms of just declaring not declaring, but analyzing whether it has worked or not. I've probably been not patient enough. Certainly, you see, oh, here's a next generation architecture, and maybe none of that or very little can be imported into China than that that would seem to to have a potentially quite large impact in the coming years. What is China gonna do about that? Obviously, there are very impressive companies in China. We've mentioned Huawei earlier. What does the attitude seem to be there? Is it like we're gonna I've seen a time lapse of a hospital going up in 6 days in China in a pinch during the beginning of the pandemic. Do you think that same can do attitude prevails, or is there a sense that this is really not a technology deficit that can be overcome domestically?

L-squared (1:06:44) Yeah. I think there's probably mixed views on this in China. Definitely, you see the rhetoric around China needing to indigenize its semi ecosystem and supply chains, and the government is willing to throw a lot of resources at that effort. I think Huawei is really the main hope for Chinese chips. It's worked with SMIC fab in China to produce a chip that really surprised a lot of observers last year, a smartphone chip, and they are looking to expand GPU production this year. And so there are, you know, lot of hopes resting on them, but also challenges in terms of increasing yields. And it's really not an easy task to catch up to the leading edge in this super complicated industry. Yeah, in summary, people are working on it trying very hard, but you can't just will success to happen in this area. It's a very tricky task.

Nathan Labenz (1:07:56) Yeah. Interesting. 1 bright spot, I wonder if you have a a read on this. Well, a bright relative bright spot in US China relationship has been Microsoft. As you mentioned, like, you can be a Microsoft customer in China, which I don't think you can really be a a customer of Google in the same way. Maybe you could get Google Cloud. I don't know if you could get Google Cloud or AWS in the same way you could get Azure. Correct me if I'm wrong, but it seems like Microsoft certainly has a unusually good working relationship with China and that even in the research where there are a number of great papers that have been US China collaborations mediated through Microsoft. Often, it's like Microsoft Research and Xinghua. Do you have any sense for, like, how Microsoft has managed to keep such warm relationship amidst general increasing tensions?

L-squared (1:08:47) Yeah. Good question. You mentioned other companies there. AWS does have a presence in China. It has couple of data center regions. So Microsoft's not you know, the only company with a presence that is quite unique in terms of having such a significant research presence because a lot of the other American tech companies that do have a foothold in the China market would still be doing that R and D outside of China. So what makes Microsoft different? I think for a long time, there's just been this recognition from Microsoft leadership that there's a lot of great research talent in China that they want to tap into. I think going back a number of years now, they've been investing in developing that talent community and many top leaders in Chinese AI have been trained up at Microsoft Research Asia. There's some interesting pieces that Jeff Ding and his China AI newsletter have translated on the history of Microsoft Research Asia that I could refer listeners to. But I would say that it is getting increasingly hard to be a company that's trying to maintain good relations with both the Chinese and the US governments. I think you've already seen Microsoft get questioned from US politicians about its research presence in China. There were some rumors last year that it might be thinking about moving some of its research scientists to Canada from Beijing. So there's still a bit of a question mark about how long Microsoft can maintain this position of tapping into top research talent in China.

Nathan Labenz (1:10:30) I wish them luck in maintaining their positive relations, and I definitely have learned a couple notable tidbits here, including that AWS has a presence there. I did not know that. So that that's definitely a update to my worldview that there are multiple big tech companies doing nontrivial business in China. On the collaboration at the sort of academic scientist level, this is also a confusing thing for me. I I don't know if you have a theory of this. I read archive papers daily, and a lot of the names are Chinese. The authors' names are Chinese. And often, all the authors' names are Chinese, and often it's a Chinese institution that they were working out of. So I'm surprised that given all this sort of general ratcheting of tensions, there hasn't been any sort of pullback from China in terms of, hey. We're we because we've certainly seen this from the leading frontier model developers in The US not publishing their methods anymore. The Chinese research community, though, seems to not just continue to publish, but continue to publish in American venues in English a lot of the time. And I guess I wonder why is that? Is that just scientists rising above political discord to continue to work together, or is there something else going on there that keeps the flow of research on out out of China and to the rest of the world?

L-squared (1:11:58) Yeah. I would say that China is keen. Chinese government is keen for its researchers, its academics to be reaching a a global audience. It looks good for Chinese power if it's rising up the ranks in in terms of contributions that scientists are making in the world. So you see that reflected in the incentive system that researchers in China are working within. They are incentivized to produce as many publications as possible, and that means publishing in top English language journals, presenting at top international conferences. And there are some Chinese journals, but most top Chinese researchers would not be bothering with those because you're just cutting yourself off from a larger potential audience, then you're not gonna be rewarded as much for it in the academic system.

Nathan Labenz (1:12:55) So would I infer from that that there are no Chinese versions of the papers that I'm reading that are coming out of Chinese institutions? Yeah. They're, like, literally only publishing in English. Wow. That's fascinating. Is does this fall under, like, face, the broad concept of in saving face? Is that what this is, or is that not the right way to think about this?

L-squared (1:13:17) I'm not sure. I think there's a risk of trying to see too much through this face lens. I think it's understandable at both an individual level and a government level that the researchers themselves want to have a larger impact to be recognised by more people, publish in the best venues possible. And also at the government level, China wants to be recognised as a international science hour, then this is where they've got to be publishing. That's just how international science is, and it's dominated by English at the moment. So I think it's a case of Chinese researchers working within the current power structures.

Nathan Labenz (1:13:59) Long may it continue. I hope the the scientific collaboration avoids getting disrupted in the way so many other ties between the 2 countries have. It is funny. I heard Jordan Schneider's talk report from NeurIPS on the China Talk podcast, and he said that there are just a remarkable number of people speaking Mandarin at NeurIPS. I believe this last 1 was in New Orleans. And it's visa issues, all that kind of stuff aside. Like, you just walk through the halls there, and a good chunk of the conversations are happening in Mandarin. So it is very, very interesting to think about that juxtaposition of, like, there's enough critical mass that people can speak their native Chinese language. And at the same time, they're all even out of Chinese universities still publishing in in English. It's a weird world. So I guess last couple questions, and I really appreciate your time. You've been super generous with it. The the big picture, obviously, is this rivalry between the 2 countries. 1 article that I have not been able to get out of my this is like my Roman empire arguably with these days. An op ed in the Washington Post from May 2018. This is before GPT 2, which is early 20 19. The article written by a person named Feng Zhang, you can tell me if I'm saying that wrong, professor at Tsinghua, 1 of described by the Washington Post as 1 of China's most prominent legal scholars. The article is titled AI will spell the end of capitalism. That is almost 6 years old prior to language models, but definitely a pretty provocative headline even by today's standards, maybe more so today just given how much more powerful AI systems have become. The notion there is basically the market has been the only way to figure things out, but, like, AI maybe will give us a different way to figure things out, and China's gonna lead on this. And there's sort of a prospect for a return to a more planned economy, a more planned and controlled society. Is that kind of the line that is there a government line domestically in China? And is that sort of what it is? Like, we're gonna use AI to get even more communist, more more sort of centralized planning, more centralized control, or is there a different narrative that is heard from the government?

L-squared (1:16:15) I wouldn't say that article really reflects the government position. I do think that the government sees a lot of opportunity in AI, sees that it can help with industrial optimization and productivity gains, but I haven't seen it spread the message that it's gonna lead to some communist utopia. There's also risks that the government recognizes and has talked about in terms of exacerbating social inequality and disrupting employment structures. I wouldn't say that it's as optimistic as that article is. And a lot of the opportunities and risks that it talks about from AI are similar to the ones that you see other governments talking about.

Nathan Labenz (1:17:03) Interesting. Yeah. I've heard you say that if you of course, through this conversation, and it has rung in my ear as, I guess, just another kind of uncanny similarity. It's not given the fundamentally new and weird nature and powerful nature of this technology. It makes sense on some level that people should have a similar attitude toward it, but it is striking and surprising that basically the line seems to be the same. Get the upsides, minimize the risks, and that's kind of it. And it sounds like not a lot more detail there either. This is something I've been really lamenting recently about the Western discourse is just that we don't really have a positive vision for the future. And I don't necessarily think it's the government's place to offer that here, but even, like, industry leaders are not really offering much in the way of a concrete vision for a positive future. It's all just like, it's gonna be amazing as long as we don't blow ourselves up in the process. But it sounds like that's basically the same thing that is prevailing in China. Could be could be awesome, and hopefully, we'll get there. Is that is it really that kind of lacking in in detail?

L-squared (1:18:07) I would say that similarly, there isn't really concrete, clear long term vision for AI and how it'll be integrated into the economy. This year, you had the recent publication of the government work report where it set out its priorities for year, and AI was mentioned fairly prominently there. And they were talking about this AI plus initiative, which seems to be about integrating AI with more different industries and sectors and really trying to to bring forward more real life application to different sectors. But, yeah, that's not really the same as as deep thinking about how we're gonna, you know, be using AI responding to AI when these systems get way more powerful in the future. Yeah. I think a lot more thinking to be done on that topic.

Nathan Labenz (1:19:05) Yeah. It sounds like that's a there's a global shortage of positive vision for the future. 1 thing we do hear often in about China and I guess maybe like Asian cultures more generally is that there is a higher default level of enthusiasm for or optimism about technology here in certainly in The US, I would say probably Europe too, you have much more skepticism, much more like, it's gonna take all our jobs, and then it's gonna suck or whatever. It's not to mention the Terminator memes. But would you say it it does seem to be true that there is more just day to day man on the street sort of optimism about a technology future in China?

L-squared (1:19:47) Yeah. I think the more positive attitude you're referring to does seem to come through in opinion polls that I've seen. In the West, you see NGOs playing an important role in raising concerns about certain abuses of technology. And you just don't have that kind of active NGO landscape here because of many restrictions that nonprofits face. You don't have these organizations that can really mobilize and raise concerns about tech in quite the same way, although sometimes you do see important critical pieces from journalists. Yeah, my take would be probably a bit more optimism, but also shaped by the fact that you don't have the same kind of open space for for discourse and for diverse voices that you have in the West.

Nathan Labenz (1:20:41) In the context of self driving cars, is there a because that's another 1 here where and it drives me nuts. There there's just a lot of pessimism and negativity. It's either it'll never work or I don't want it even if it does. And I have no idea what the situation might be in China, but if I had to bet on which country is gonna get self driving cars first, I would probably bet on China. And the reason is actually less to do about the attitude of positive or negativity, but more just if something like that is feasible, I would expect that the Chinese government would be like, hey. This is gonna make life safer and better and people can work in the commute or whatever. So let's do it and actually make a point to make it happen. Whereas here, certainly, the authorities are, if anything, standing in the way and very gradually allowing stuff to happen. But they haven't been totally blocking it, which has been a pleasant surprise, but it certainly is not something that the government is, like, trying to make happen. So on the self driving car front, do you see that as is there any indication that the government wants to bring that reality to the public?

L-squared (1:21:47) Yeah. There is a positive attitude from the government towards self driving cars, and there are companies, including Baidu, that has been testing cars in suburbs of major cities. In parts of Beijing. There are now taxi trips that you can take without a safety driver, but they are limited to certain routes, certain sets of destinations. I think that what you say about government potentially being more willing here to provide support could be true. And I think the fact that Chinese consumers are often quite open to trying new technologies can also be a point in favor of the Chinese self driving car industry. I would add though that the city centers in China can be pretty chaotic. Got loads of pike lists and delivery drivers going the wrong way. No 1 really following the road rules. I think that could be a big obstacle, at least for very busy, messy city centers. But certainly in in more straightforward environments, I could see adoption being pretty impressive and and fast in China once the technology is good enough.

Nathan Labenz (1:23:00) Is that something where you could see the government actually transforming the landscape to take advantage of it? We have this picture of China with the high speed trains and whatever, and it's like, I I heard somebody say 1 time, you'll never see a straighter railroad than the 1 from the Beijing Airport to downtown. I don't know if that's true or not, but the idea was simply that somebody draws a line on the map, and that's where it's gonna go. And there's not a lot of doing and blocking and this and that. And, obviously, there are pros and cons to such decisive moves to build. Is it does it seem conceivable that you could see, like, these sort of chaotic city streets tamed in some way for the purpose of ushering in a new technology regime like a self driving car, or is that just, like, too crazy to contemplate from where you're sitting today?

L-squared (1:23:49) Yeah. Good question. Definitely not outside the realm of possibility. You do see already Chinese government being willing to invest in new cities and new infrastructure to support self driving cars. So there's place called Xiong'an that's not too far away from Beijing and in a neighboring province built as this new town and has lanes specially for driverless cars. There's a expressway linking Beijing to this place in Hebei and that expressway is being designed with self driving cars in mind. I think that is a positive signal. It's obviously harder to completely rework existing really old cities, but the the Chinese government can do pretty impressive things when it puts its mind to it, as you've alluded to with the mention of the train system.

Nathan Labenz (1:24:40) Cool. That's that's a another great little tidbit, the idea that there's a a highway under construction with self driving cars in mind. That's exactly the kind of thing I was thinking might might happen there, and I don't see that happening here in the near term. But, hey, maybe we'll be motivated by not wanting to allow a self driving car gap. If so, that'd the kind of technology race I would encourage. Going back to I think probably the last big question I have for you is just around the culture and thinking of AI safety in China. I think, again, 1 of the very annoying things to me that is often said in the West is that we can't trust China to cooperate with us in any way, shape, or form on AI safety. Therefore, we just have to develop the technology as fast as possible because otherwise, will. And I don't really buy that, but I also don't know a lot about what people in China do think about questions of AI safety. I know we've seen some joint statements recently, which is seemingly encouraging that there is some room for cooperation. And there's even, like, a governmental agreement, at least in principle, to not use AI systems to launch nuclear weapons, which is doing the minimum. But, hey. It's great. We should celebrate that, I would say. And beyond that, I just don't know. What would you say is the the rough shape of thinking in China when it comes to big picture questions of AI safety?

L-squared (1:26:06) Yeah. Firstly, not quite sure which agreement you're referring to on the nuclear side of things.

Nathan Labenz (1:26:13) Yeah. Let me look it up real quick. 11/11/2023, Biden g set to pledge ban on AI and autonomous weapons like drones, nuclear warhead control. This was out of that meeting in San Francisco. And I

L-squared (1:26:32) think there was speculation before Biden and c met in November that they might announce this agreement to use AI and nuclear systems. But when the actual readout came through from the White House, it just had a brief line on AI, which said that Biden and c agreed that there would be future dialogues on AI risk.

Nathan Labenz (1:26:55) Yeah. As I'm reading all these headlines, I think you might be right. Unfortunately, the all of the sort of early and mid November articles say set 2 and 2 sign, and then an article on December 4, couple weeks later from the lawfair.org, China won't yet commit to keep autonomy out of its nuclear command and control. It will take a lot more talking to get there. Yes. That's been perhaps deferred to future discussions, which is a bummer because I thought we had agreed I thought we had agreed to not have AI systems in our nuclear command and control. So, yeah, I won't sleep quite as well tonight. But I guess the big good call out on that, because I had bought into the perspective and not yet actually confirmed headlines about that agreement, with the head backdrop of that agreement was not in fact actually reached, and I'm somewhat reeling from that realization. Give me some tell me the truth, but I'm hoping for some good news on the big picture questions of AI safety in China.

L-squared (1:28:01) Yeah. Sorry to disappoint you on that 1. I would say that there have been some positive signs in the past year or so. We've seen more dialogue between Chinese and Western actors around AI safety. In June, there was a whole day of talks on AI safety and alignment conference held by the Beijing Academy of AI, and Sam Altman delivered an online talk for that, and many other prominent Western AI figures spoke as well. We also had the AI safety summit in The UK that was attended by the Chinese government and other Chinese individuals, and it was great to see agreements coming out of that on catastrophic risks from AI. On the other side, I'd say that there is still a gap between and and concrete actions, and you don't see as much research happening in China on AI alignment. You don't see top labs here committing to responsible scaling policies in the same way that Anthropic and OpenAI have. There's a lot less thinking here about corporate governance mechanisms that might need to be in place to mitigate the worst risk from AI. But maybe this is because you don't have in China such a long history of labs that are explicitly focused on building AGI, pushing the cutting edge of the technology quite the same way. Maybe it's understandable that Chinese developers are just focusing on trying to improve their tech and get up to the level where they can really be following the likes of OpenAI and Anthropic and they're not really putting so much energy into thinking about safety. But I'm hopeful that with continued dialogue with Western actors and academic circles and and at the government level, you could see movement on this front and more actual action and investment as opposed to just talking.

Nathan Labenz (1:30:16) My last question, is there anything that you think folks in the West could do to encourage that continuation of dialogue and and hopefully some big picture cooperation as opposed to the what everyone fears, I think, which is the AI arms race between the 2 countries. That sort of at the level of what could governments do, but also what could you know, what what should I advocate for starters, if anything?

L-squared (1:30:47) Thanks for that question. I think that for people who have influence within AI circles, within policy circles, just trying to keep channels of communication open such that there can still be room for dialogue on safety and risk reduction. I think the general members of the public, maybe people who don't have the era of government in quite the same way, I think there's still an encouragement that I would make to try and engage mindfully in discussions about China and engage critically with the content that you see. I think it's important to just recognize that it is very hard to get a good handle on what is happening in China. You don't have a big community of foreign journalists here. They're stretched very thinly. And a lot of the English language discussion about China and Chinese AI can lack nuance. So encourage people to take stories that they see with a pinch of soul to try and read people who are talking about these issues in a thoughtful way, recommend the work of people like Jeff Ding and Sheehan on the AI regulation side in particular, and trying not to spread overly exaggerated views of Chinese AI. I can get frustrated when I see what really seems to be spearmongering, this view that China is a real threat to AI leadership on The US side, and we've got to do everything we can to stop it. I hope that there can be more nuanced and thoughtful discussion about the pros and cons of engaging with China and more considered approach to these questions.

Nathan Labenz (1:32:32) I'd like to think that you and I have done our small part this evening and morning to advance that agenda. I really appreciate your time, and I've learned a lot from this conversation. Is there anything else you wanna touch on before we break?

L-squared (1:32:48) I don't think so. Maybe we can put some links to some of the experts that I mentioned with the the resources that I think could be useful if people wanna do a bit more digging. But, yeah, just finished by thanking you a lot for giving me the chance to share my thoughts today, and it's been a pleasure.

Nathan Labenz (1:33:07) Pleasure's all mine. L squared, you can find her writing on the China Talk blog from time to time. Thank you for being part of the cognitive revolution.

L-squared (1:33:16) Thanks, Nathan.

Nathan Labenz (1:33:18) It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcrturpentine dot co, or you can DM me on the social media platform of your choice.