Welcome to AI in the AM: RL for EE, Oversight w/out Nationalization, & the first AI-Run Retail Store

Hello, and welcome back to the Cognitive Revolution, or … in this case I should say … Welcome to AI in the AM.

This is the third time that my friend Prakash Narayanan and I have done a live stream together, and this time we figured we should give it a name, and he also took the initiative to create a new look for the show, with real-time AI transcription and AI-powered comment moderation. Check out the video and I think you'll agree that he's done a really nice job with the look and feel.

As you'll see, in some ways we are very much still figuring out both what we want the show to be, and how best to organize and produce it. One thing we're going to look at doing after this episode is creating a mechanism where we can easily signal to one another when we'd like to ask a follow up question or move on to another topic.

But when it comes to the quality of guests and conversations, I think this episode is right where we want to be.

Our guests for this episode were:

Sergiy Nesterenko, CEO of Quilter, which is using Reinforcement Learning to train AI systems to perform circuit board design, a problem with an insanely high-dimension search space, complicated physical constraints, and relatively low volume of available training data.

Andy Hall, Professor of Political Economy at Stanford, who's doing a bunch of interesting work to characterize models' behavior in political contexts, and who is also working to design independent AI governing bodies that he hopes will allow the public to exercise some oversight over AI companies without requiring nationalization.

And then finally, Lukas Peterson and Axel Backlund from Andon Labs. You may know them from their autonomous vending machine work, but today we'll be talking about the new AI-operated retail store they've recently opened on Union Street in San Francisco. The store, which is managed entirely, including hiring human staff, by an AI agent, currently has a 2.6 star rating, but I still can't wait to visit.

For me, the big takeaway from this series of conversations is once again that the future is coming at us much faster than we can process it. Assumptions that seem safe from one perspective become very questionable in the face of increasingly powerful & autonomous AI systems.

And with that in mind, I want to add just a bit to my answer to the very first question Prakash asked me in our opening discussion, namely: why are we now suddenly seeing violent outbursts directed at AI lab leaders?

First of all, while it’s certainly possible – and I would very much hope – that the recent attacks on Sam Altman’s home will ultimately prove to be a random blip, signifying nothing … my honest assessment is that by default we should expect to see more of this sort of thing - and not because of the super high p(doom) numbers coming from the AI opposition camp, but simply due to the fact that more and more people are now becoming aware of the extreme reality of the AI situation.

Sam, Dario, Demis, and Ilya all signed, alongside many other luminaries, a statement saying that “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war”

And though the record does show that each of the leading AI companies was founded with awareness of an intention to address the hard problems of AI safety, and of course the upside is unquestionably immense, in practice today they are developing what they recognize to be destabilizing and likely dangerous technology, pretty much as fast as they possibly can, while repeatedly failing to live up to their own prior safety and social commitments, in significant part because – and here I am quoting Sam Altman's immediate reflections after the Molotov cocktail incident – "being the one to control AGI" has a "ring of power dynamic to it."

All while, by their own accounts, we still face anywhere from a 5-20% chance of something like, as Sam himself famously put it, “lights out for all of us”, and the US government’s main concern seems to be making sure that nobody can constrain their ability to use the technology for autonomous weapons, domestic surveillance, or anything else.

I can’t emphasize enough: objectively, this really is a crazy situation. Those of us who stumbled onto the idea that all this might happen years ago have had a lot of time to get accustomed to it and to position ourselves to do what we think we can about it, and many have reconciled ourselves to the idea that some version of it is inevitable, but that doesn't mean we should try to tell people who are only learning about this now that they are wrong for freaking out about it. A 1 in 20 chance of human extinction is not low, and absolutely is worth freaking out about.

I’m reminded of 2 movies that memorably illustrate how I think many people will respond to learning the facts about AI.

In the 1998 movie Armageddon, when an asteroid is found to be on course to destroy the earth, it’s just understood that it's heroic for individuals to risk - and in the end even sacrifice their own lives - to save the world. And that doesn’t hinge on whether there’s a 5% or 20% or 99.9% chance that the asteroid will really hit the earth - the heroes would be heroes in any case.

In contrast, in the more recent “Don’t Look Up”, the main characters are just continually frustrated that nobody can be bothered to recognize the crisis, which drives them to become crazier and more desperate until … spoiler: everyone does ultimately die in the end.

I would submit that the difference between a hunger strike and an act of violence is not about how one understands the stakes or the odds, and neither is it about the impulse to martyrdom - rather, the difference is simply that one course of action attempts to call others to a higher ethical standard, while the other is condemned by every principled moral tradition, and even on purely consequentialist terms, seems almost certain to make everything harder and worse.

To the AI opposition movement - which fwiw I think is increasingly distinct from people focused on AI safety - I would say … absolutely continue to condemn violence, but at the same time be careful not to shy away from the fact that it’s the situation that’s crazy, not the people who are desperately searching for ways to make a difference. Your job, in addition to educating people about the reality, as you see it, and as the lab leaders themselves have described it, is to identify and create productive ways for people to act heroically in this moment. Those could include mobilizing voters to contact elected officials and advocate for regulation and international treaties, investing in citizen-level diplomacy with China, developing new governance models, pursuing experimental technical alignment strategies, and importantly, probably lots more that nobody has thought of yet. I personally always encourage people to pursue their own AI safety ideas, however eccentric they may seem, in the hope that some of them might actually pay off, and because I believe that in the absence of constructive ways to devote oneself to the cause, we will see more people simply going crazy.

As always, I will welcome your feedback – both on this analysis, and on the new show format.

Until further notice we do intend to run these conversations on The Cognitive Revolution feed, but if it goes well, we might spin it off into its own thing. If you'd like to see that happen, we definitely encourage you to follow the new show account on Twitter @AI_in_the_AM, and watch out for the next live stream, which is currently planned for April 20 at 11:45am ET / 8:45am PT.

Thanks for being a part of the Cognitive Revolution, and now, one with the show.

Watch now!

Thank you for being part of The Cognitive Revolution,
Nathan Labenz