Latest news
Episode

Zero to One in AI Safety: Halcyon's Mike McCormick on Launching 30 New Orgs & the Founder Bottleneck

Halcyon founder Mike McCormick discusses the shortage of experienced founders in AI safety and biosecurity, detailing how Halcyon incubates new organizations and builds essential institutional capacity across interpretability, verification, and governance.

Nathan Labenz
Zero to One in AI Safety: Halcyon's Mike McCormick on Launching 30 New Orgs & the Founder Bottleneck
81 min read

Watch Episode Here

Listen to Episode Here

Show Notes

ℹ️
The following notes are AI-generated, based on the episode transcript. Please listen to the episode for the full conversation.

Building the organizations AI safety needs

What does AI safety need most: more money, more ideas, or more people who know how to build an organization? For Mike McCormick, founder and CEO of Halcyon, the missing ingredient is experienced founders and leaders. The nonprofit Halcyon Futures and venture fund Halcyon Ventures support people making the move into AI safety, cybersecurity, and biosecurity, often before they have settled on a project. Mike says that in roughly three years, Halcyon has helped launch about 30 organizations that have collectively raised around half a billion dollars. His central concern is that future funding could arrive faster than the field can create excellent organizations capable of putting it to work.

The story of Goodfire makes that approach concrete. Mike describes making Halcyon’s first two grants to Eric Ho and Dan Balsam so they could cover their living expenses while deciding what to do in AI safety. A retreat, introductions, and a decision to focus on interpretability helped turn that exploration into a company; venture investment followed. Nathan discloses his own small early investment in Goodfire. The conversation then turns to AIUC, whose combination of the AIUC-1 standard and insurance aims to give companies commercial reasons to reduce AI risk. Mike uses ElevenLabs as an example of how an AI vendor could make its products easier for enterprise customers to adopt by pairing safety assurances with coverage.

The range of possible work is striking. Transluce studies scalable oversight and builds tools such as Docent to help researchers understand increasingly capable models. Hadrian Biodefense tackles pandemic preparedness, including the protective equipment essential workers would need during a disaster. Mike also describes work on indoor air quality without naming the company involved. Nathan’s attempt to get UVC lights installed in a school classroom brings the adoption problem down to earth: even a willing parent can struggle to find someone authorized to approve a new intervention. Mike sees schools as promising early customers because cleaner air could reduce ordinary illness and absences as well as help during a future pandemic.

Can new institutions become useful quickly enough if something like the AI 2027 scenario unfolds? Mike is uncertain about timelines, but prioritizes projects that could matter within the next one to four years. He wants more independent capacity across interpretability, control, evaluation, cybersecurity, and verification, pointing to METR as an example of valuable work that occupies only part of a much larger problem. Verification is especially important to his account of how agreements could become enforceable: being able to establish which model is serving a request, or whether a data center is being used for training rather than inference, could give technical substance to commitments. Software, cryptography, hardware, and coordination all have to develop together; a technical system without an agreement to use it, or an agreement without a way to check compliance, leaves a major gap.

Nathan also presses on the relationship between mission and financial incentives. In a general discussion of organizational governance, Mike points to Mission Preservation Governance Mechanisms for Startups and argues that public-benefit structures and board mechanisms can help, while none can guarantee how leaders will act under pressure. Founder motivation and a record of execution remain central to his judgment. The broader funding discussion touches on Coefficient Giving and its Project Tailwind, the Survival and Flourishing Fund, and the OpenAI Foundation. Mike’s emphasis is on the work around an organization’s creation—helping people choose a problem, find cofounders, recruit a team, and build an initial offering—so that later funders have strong projects to support.

Several harder-to-place ideas test how broad that mandate can be. Nathan highlights AE Studio’s Self-Other Overlap research and GRAM, asking whether training methods could make models less deceptive or restrict dangerous capabilities in released weights. Mike remains interested but cautious about how safety measures survive malicious use of open models. Their discussion of consciousness distinguishes uncertainty about whether models have experiences from risks that do not require consciousness at all. Eleos AI Research, Apollo Research, Softmax, and Reciprocal Research enter the conversation as reference points for questions about welfare, agency, and how humans should relate to increasingly capable systems. Both speakers leave room for uncertainty rather than treating those questions as settled.

The institutional challenge returns in the discussion of public communication and coordination. Mike describes the Seismic Foundation’s work on communication about the AI frontier and points to the AI Evaluator Forum, the Frontier Model Forum, and Fathom’s proposal for independent verification organizations as different pieces of the emerging landscape. Nathan draws on Polis and Audrey Tang’s work in Taiwan to ask how people can move from an idea to an institution with real authority. Mike is interested in new democratic and coordination tools, but stresses that a compelling technical design still needs a viable route to adoption.

The episode closes with a practical invitation to experienced people who want to contribute. Mike sees particular demand for founders, COOs, recruiters, cybersecurity professionals, cryptographers, formal-methods specialists, and mathematicians. He argues that organizations should pay enough for people to build sustainable lives, while acknowledging that some career transitions involve a pay cut. His advice is to hustle without rushing: explore whether to found or join, compare nonprofit and for-profit options, test ideas, and find the work that fits unusually well. Listeners can explore the Halcyon Request for Founders or contact hello@halcyonfutures.org.

Topics covered

  • Halcyon’s founder-first approach and the shortage of experienced AI safety leaders
  • Goodfire’s path from career-transition grants to an interpretability company
  • AIUC: standards, insurance, and incentives to reduce risk
  • Transluce, scalable oversight, and tools for understanding model behavior
  • Hadrian, pandemic preparedness, and the practical challenge of adopting cleaner-air technology
  • Short AI timelines, pacing, and the verification tools that credible agreements require
  • Choosing whether to found or join; career grants and founder–problem fit
  • General mission-preserving governance, founder motivation, and financial incentives
  • The funding landscape and why future philanthropists need strong organizations to fund
  • Open-weight model safety, Self-Other Overlap, and GRAM
  • AI consciousness, moral patienthood, and risks that do not depend on sentience
  • Public communication, independent verification, and turning proposals into institutions
  • Alternative AI architectures and the difficulty of proving both capability and safety
  • Career transitions, compensation, and the operators and specialists the field needs

Resources

Quotes worth pulling

The field is hurting for world-class founders who can take those big squishy ideas and turn them into organizations that actually make the world safer.

— Mike McCormick

And I find that so many of society's stickiest problems are not technical problems, they're coordination problems.

— Mike McCormick

And I think basically none of the risks that I care about require consciousness.

— Mike McCormick

What is as important as the contents of the Constitution is, how are we gonna go from this thing being some stuff we wrote down on a piece of paper to an actual basis for government?

— Nathan

But don't just rush into the first shiny object — the best thing you can do is orders of magnitude better than the median thing, or even a pretty good thing, whether you're starting something or joining something.

— Mike McCormick

Sponsors:

ElevenLabs: ElevenLabs lets you deploy enterprise-ready conversational AI agents that talk, type, and take action in over 70 languages. Schedule your demo today at https://elevenlabs.io/tcr

OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr

Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

CHAPTERS:

(00:00) About the Episode

(03:54) Introducing Halcyon

(06:09) AI safety portfolio stories (Part 1)

(19:07) Sponsors: ElevenLabs | OutSystems

(21:57) AI safety portfolio stories (Part 2)

(22:42) Halcyon's core thesis

(34:14) Funding vs joining orgs (Part 1)

(34:19) Sponsor: Claude

(35:54) Funding vs joining orgs (Part 2)

(47:03) Mission preservation governance

(01:00:21) Neglected safety areas

(01:09:09) Evaluating AI consciousness

(01:19:36) Institutional coordination mechanisms

(01:35:59) Transitioning into AI safety

(01:46:11) Episode Outro

(01:48:08) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution.
Today, my guest is Mike McCormick, founder and CEO of Halcyon, a combination nonprofit grant maker and for-profit venture fund working toward a single mission of helping the world's most talented founders and leaders launch new AI safety, cybersecurity, and biosecurity organizations. Mike's core belief is that organizations solve problems for the world, and of course, AI is creating a lot of problems that we need to solve in a remarkably short period of time, which means that what we need most right now isn't more ideas, but founders who can build the organizations needed to implement whatever turn out to be the most viable solutions at scale. In three years, Halcyon has helped launch roughly 30 such organizations, which have collectively raised roughly half a billion dollars. Goodfire, who's been featured on the show multiple times, is the best illustration of how Mike works, with a special focus on the six months before and six months after an organization is started. The first career transition grants Mike ever made were to eventual co-founders Eric Ho and Dan Balsam while they were still running their previous company, but starting to think about changing focus to work on AI directly. The financial boost, the vote of confidence, and the network building support he provided helped them found a company which has since achieved unicorn status and become one of the best places in the world to do interpretability research. Other orgs that Mike has supported include past guests AIUC, which does agent standards and audits for insurance purposes, Fathom, which has advanced policy discussions with credible early proposals for independent regulatory schemes, and Asymmetric Security, which is building cybersecurity forensics agents meant to defend companies from employee account compromise incidents. Beyond that, Mike also introduces Transluce, which is working on scalable oversight, and Hadrian, which is stockpiling PPE. It is admittedly a very broad portfolio, but as Mike puts it, we are speed running six or seven industries worth of AI revolution all at once. In a twenty twenty-six that remains plausibly on the AI twenty twenty-seven timeline, rapid, competent execution is obviously critical and clearly will require heroic efforts. To take just one personal example, for nearly a month now, I've been trying to donate Aerolamp lights to my son's classrooms. And while my son's teacher said that she loves the idea, it's a new thing, and nobody at the school has been able or willing to sign off on it. This too, it turns out, Mike is involved in starting an organization to address. Naturally, we talk about how seemingly short timelines affect his strategy, what he looks for in people to back, and how he thinks about both for-profit and nonprofit governance. And we also discuss the verification technologies we'll need to sustain pacing agreements, as well as the challenge of getting everything from cryptography to hardware manufacturing to diplomacy all working together as one. The main headline of today's episode, though, is that there is still time for ambitious leaders to start new AI safety organizations which could prove incredibly important. And more broadly, that there's never been more support to scale these organizations with first class talent, which means that all sorts of accomplished professionals, including many who do not have technical ML backgrounds, should be actively thinking about pivoting their careers into AI safety. If you are feeling the pull, Mike offers his contact info later in the episode, noting that a warm introduction is welcome but not required. With that, I hope you're inspired to see yourself as a potential difference maker by this conversation about building the institutions that we will need to sustain AI safety with Mike McCormick, founder and CEO of Halcyon.

Main Episode

[03:54] Nathan Labenz: Mike McCormick, founder and CEO of Halcyon. Welcome to the cognitive revolution.

[03:59] Mike McCormick: Yeah. So good to be here. Thanks for having me.

[04:02] Nathan Labenz: Thank you. This has been a long time coming. I think we met a couple years ago now, and you have been one of the behind the scenes movers and shakers that have been pretty instrumental in in creating and getting off the ground a bunch of different organizations that are now active across the AI safety space. And I think I can speak for everyone when I say thanks to you and to others who've done similar work because, this is all coming at us pretty fast. And lord knows we need, all the different organizations and specialists that we can that we can possibly have to navigate this, critical time that we find ourselves in. Let's start maybe by just kind of introducing, like, what is Halcyon? There's a couple parts to it, and you got and then then we can, of course, move into talking about some of the organizations that you've helped to stand up.

[04:53] Mike McCormick: Sure. So, yeah, Halcyon, we're a three year old organization. So we are part nonprofit and grant making organization, and then we are part venture capital funds. So we have two arms to the organization, but don't think of them as two sort of separate projects. Think of them as, you know, two separate tools in in the same bag. Right? And so our whole mission is to find the world's most talented leaders and founders and basically help them, start and launch new AI safety organizations, cybersecurity organizations, biosecurity organizations. And so in three years, we've launched, like, 30 new things, and those things have collectively raised, like, a $500,000,000 in funding. And, again, some of that funding is is nonprofit, sort of philanthropic funding. Some of it is is venture funding. And, yeah, I think you've had, founders of of a few of these organizations, like Goodfire and AIUC, and I think a few others too.

[05:52] Nathan Labenz: Well, let's let's do a little deeper dive on just some of the organizations in particular. I mean, you know, people can go listen to the full episodes with Goodfire, with AIUC, Fathom, and Asymmetric. But give us the kind of quick pitch on, either those or, you know, whichever ones are kind of top of mind you think people should be aware of.

[06:10] Mike McCormick: Well, tell some stories. I mean, Goodfire is a great a great a great one. So I met Eric Ho and and Dan Balsam, who are the founders of the company two or three years ago when they were still running their last company. It was a pretty successful tech company. Although they they were starting to think a ton about, like, AI safety and AI risk and, like, what what can we go do? And so, actually, the first two grants Halcyon ever made were career transition grants to Dan and Eric so that they could just go, you know, pay their rent while they they figured out what to do in AI safety. And then we brought them to a retreat and connected them with a bunch of great people there up in the woods in Northern California. They landed on interpretability as this sort of central problem that they wanted to work on and ended up recruiting a couple other great people and and launching Goodfire. And then from our venture capital funds, we we were among their first investors and invested in every every round since. Just chatted with Eric for a minute on the phone yesterday. And so, you know, I think what Goodfire is is is is the interpretability lap. Right? Like, one would think that, you know, if we're if we're gonna understand AI models and we're gonna understand all of the ways that they might be dangerous or deceptive or misaligned or doing things that we don't want, you know, you would be really great to be able to sort of look inside those models and say, ah, okay. These neurons are firing in these ways to produce sort of this dangerous output. Okay. Maybe we can do something about that. Right? And so and so that's that's what what Goodfire does. I think they're they're probably the best place in the world to, work on interpretability research, even including the labs. Although, you know, perhaps Anthropic could could give them a run for their money. And so they've they've become a pretty big company. They they recently raised a a a I think it was a well over a $100,000,000 series b above a a billion dollar valuation. And, yeah, just been super impressed with their ability to, you know, make progress in in the interp space.

[08:12] Nathan Labenz: Yeah. I love doing episodes on their work, and it seems like they've been so prolific that every time I go back, I'm like, holy moly. I've got, a dozen papers I've gotta catch up on, and it's only been, you know, sixty, ninety days each time. So

[08:27] Mike McCormick: I think, like, I think that's what a lot of companies in this space that do really well, and nonprofits, by the way, have in common, which is, like, they do a great job at at getting a bunch of the best talent. Right? And then, like, you know, the first really talented five people come in, and then, like, the next few people are like, oh, like, all my smartest friends are going there. Let's go there. And then the funders see that. And, of course, the funders wanna fund places that, you know, have really great talent and so then they pile in and then the the flywheel sort of spins. And so, yeah, I think I think they've been particularly good on that. And and, yeah. I don't know. So so much to be, to to to be excited about there.

[09:04] Nathan Labenz: There, one of my kinda answers these days for what would the value of a pacing mechanism or a slowdown or even a pause be. It's like, well, you know, look at the pace they're going, and a little more time would be really advantageous for them to, you know, potentially get over some key thresholds of understanding.

[09:23] Mike McCormick: Yeah. Totally. I mean, like, I feel like we're just speed running, like, five industries or six or seven industries at once now. Right? It's like we're speed running control, and we're speed running verification and cybersecurity and biosecurity and alignment and interp. Right? And, you know, all of these are just such young spaces. Right? Like, you know, like, they're really nascent. Right? And and and we could easily just 10 x all of them, you know, today and and still probably not have enough enough firepower behind behind solving these problems. Right? Like, you could argue that we just need, like, many Manhattan project size efforts all all at the same time here.

[10:03] Nathan Labenz: You wanna talk a little bit more about AIUC others?

[10:07] Mike McCormick: Yeah. I love AIUC. So another company, that we supported very early, also made grants to, founders before there even was a company. So the way I think about AIUC is, like, I wanna live in a world where we can, you know, measure and systematically mitigate risk from AI. And to me, that sounds like kinda what, like, an insurance market does. Right? It's like we underwrite the risk. It's, like, bad thing is happening, and then we create financial products and other incentives to, in incentivize sort of those risks or or those those bad events to not happen. Right? This is kinda what AIUC does, but for AI. So they kinda have this dual product. So one is a standard. Think of it like a SOC two standard but for AI. And if you wanna meet this standard, you have to meet these 50 something check boxes that have to do with your safety practices and your security practices and etcetera. And then along with that standard comes an insurance policy, which you can only buy if you meet the standard. And so I think it has the possibility of creating, like, a really interesting, like, race to the top as as companies, like, want to meet these standards because their customers want them to meet the standards. Right? So, like, AIUC's big customers are, like, growth stage AI application companies. Right? So, like, Eleven Labs, I think, was one of their first big com customers. And the idea is, like, Eleven Labs is selling their AI agents and tools to, to enterprises. Right? And these enterprises, like, really care about AI risk and, you know, not adopting tools that are gonna cause some, you know, huge issue for them or go off the rails or whatever it is. And so now, Eleven Labs can go to their big, you know, Fortune 1,000 customers and say, oh, hey. By the way, you can adopt us and feel really good about it because we meet this standard, this shiny AIUC one standard. And by the way, along with buying our tools, our tools are now gonna come with an insurance policy where if any of these risks that you're worried about actually come to pass, well, you're insured against them anyway. And by the way, the only way you can even get this policy is by meeting this this very high standard. And so hopefully over time, there are many more standards. I mean, think AIUC will become a major one, but perhaps not the only one. And, hopefully, these standards will become sort of harder and harder to meet and and and hopefully contribute to a little bit of a a race to the top dynamic where, like, there's actually incentive to be to be safer.

[12:44] Nathan Labenz: How about Transloose?

[12:45] Mike McCormick: I confess,

[12:47] Nathan Labenz: I've seen good things from them on the Internet, but never having done the full episode deep dive. I don't know probably as much as I should.

[12:55] Mike McCormick: Yeah. Yeah. You should have them on. Jacob would be would be great. So Transloose actually, I think Jacob came to the same retreat that you and Eric were both at as well. So Jacob was a professor at Berkeley, and his cofounder Sarah was at MIT, I believe. Wanted to launch an independent AI research organization specifically focused on scalable oversight. Right? So it's like, hey, maybe not so hard to, you know, really understand what's going on with a, you know, GPT three level model. Right? But as we get to, you know, mythos three, right, you know, how are we going to have oversight over things that are perhaps smarter than us or things that are already or or things that are just constantly, you know, maxing out their benchmarks. Right? Like, I think I heard on one of your recent shows the idea that, the the meter time graph is basically done as a useful metric because the models are improving faster than like, they can actually, like, get humans to do those tasks or, like, measure, like, how long humans can do tasks. Right? And so so how are we gonna have, you know, oversight over over models that are that are sort of smarter and more capable than us? And so that's their main line of research. And so they build tools to allow other researchers to to do that. One of their tools is called Docent. And then they also work with external organizations to do, evaluations, with labs and and with others.

[14:29] Nathan Labenz: Wanna do one more? Hadrian?

[14:30] Mike McCormick: Oh, Hadrian. Yeah. One of the big risks that I care about is is bio risk. Right? So perhaps AI makes it easier for a bad actor to create a bioweapon or to synthesize a pandemic virus. And so we've we've done a bunch in the biosecurity realm. And one thing you would really want in, let's say, like, a future COVID type disaster is just enough PPE, protective equipment, to keep essential workers going during said disaster. And so we we wrote a a tiny little check. I think it was, like, 50 k to get them off and running at this idea. They basically figured out that, you know, for on the order of a few $100,000,000 or maybe a couple $100,000,000, you could stockpile enough PPE to essentially keep every American worker, you know, at work during a COVID or perhaps significantly worse pandemic. And so that's essentially what they're doing. But I think they actually have a a broader vision than that. Right? So, like, if you think about biosecurity, like, the framework that, people tend to use is, like, deter, detect, defend, or there's other ways to say it. Right? But, like, part one, you wanna make it so these biodisasters don't happen. Part two is, like, if they are happening, you wanna be able to know and detect them very quickly. And then part three, you wanna be able to respond and sort of overcome or or or sort of get through get through those disasters. Right? And so they're doing a bunch on that on that third category. Right? Like, what are all the categories of things that society will need to sort of get through that or defeat it? And we're doing a bunch in that that they're not doing as well. So you can think about, like, medical countermeasures. Right? Like, could you spin up vaccines extremely rapidly or or non medical countermeasures? Are there other other things we can do? So for example, like can we improve the built environment? So we we just invested in a company that we're helping to launch that's basically innovating in indoor air quality. Right? So between like like air filters and far UVC and glycol vapors and other technologies, perhaps you can just like harden the entire built world to to future pandemics.

[16:51] Nathan Labenz: They have one of the better headlines I've seen on a website, help defend civilization. So I'm trying to do my small part to help defend a little corner of civilization, which is my son's elementary school classroom by Mhmm. Buying some UVC lights and hopefully getting them installed in the classroom. It's been a funny experience because the Detroit Public School that he attends, which, you know, is is honestly, he told me years ago I would send my kid to Detroit public school. I'd have said they were insane. There's no way that'll ever happen. But we now live here, there's a Montessori elementary right up the street. So it's great. But now I'm running into this bureaucracy of, like, who can approve this? And there's no department for this. You know? I'm, like, following up on email and being like, you know, really? Like, my kid had cancer. Like, I really wanna not get him sick a lot this fall. Can we please get this light installed? And they just don't know what to do with me.

[17:43] Mike McCormick: So it's It's funny you mentioned schools because schools are actually the initial, customers of of of our company here. So there's a company and there's a sister nonprofit, and the idea is that, like, schools are a place where you would, yeah, like, really like, they're they're sort of super spreader places. I'm sure you know with kids. Right? You're probably probably picking up colds and everything all the time.

[18:02] Nathan Labenz: I've already had you can hear the the remnants of my first back to school cold in my voice today.

[18:07] Mike McCormick: I think there actually is a bunch of appetite from schools to do this. Like, I think one issue with biosecurity specifically is that there isn't necessarily demand right now for a lot of the things that you would really want, say, like, in a future pandemic. Right? Like masks and vaccines, for example. But there are really good reasons to adopt air cleaning technology right now even if we never had another pandemic, like just the amount of, you know, productivity missed in the economy. Right? The amount of school days missed, you know, the amount of sick days for you, the amount of suffering from just, like, common colds and flus and bugs. Right? And so I actually think it's fairly promising that these types of technologies could be adopted really broadly even if there wasn't some, like, big pandemic freak out. So, yeah, I'm I'm more optimistic about that category than some others.

[19:03] Nathan Labenz: So we've done, like, many unpacking of four organizations, and there's a a lot more. But the first thing that I think jumps out is you've got a super broad mandate here. Right? We've gone from interpretability science to agent standards and and, you know, kinda trying to create the conditions for an insurance market to create the right incentives for people to scalable oversight nonprofit to stockpiling PPE and trying to get these measures into schools. Do you just go out and say, like, I would give me a list of every problem in the world, and, let's go stand up in order to solve them all? Or, like, how do you think about the breadth of what you're gonna take on?

[19:20]ElevenLabs: ElevenLabs lets you deploy enterprise-ready conversational AI agents that talk, type, and take action in over 70 languages. Schedule your demo today at https://elevenlabs.io/tcr

[20:51]OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr

Main Episode

[22:43] Mike McCormick: Yeah. Like, in some sense, we're super narrow as an organization. Right? Like, I think I do okay. So I I think we basically have, like, a worldview, and there's there's, like, two or three parts. Right? So part one is that AI is just progressing extremely quickly and probably will continue to progress. Right? Like, we're sorta kinda on the AI 2027 timeline, give or take, you know, a few months. Right? Thing two is that, you know, look. There's so much to look forward to with AI, of course. Right? Like, we're seeing all these amazing, you know, educational results and health care results, etcetera. So, like, certainly not anti AI. But, like, if we wanna get to this beautiful future we all want, we have to navigate the risks, like, really carefully. And the risks are, major and global in scale and, you know, intersecting. Right? And so I I am I'm I'm quite worried that we're not really on track to solve, like, the biggest pieces of the AI safety equation. And so and then I guess the third part of our worldview, which which is sort of drives Halcyon's work is the biggest bottleneck to solving this problem or to sort of speed running these fields is a shortage of really amazing founders and leaders. Right? Like, I started making grants in AI safety three or four years ago, maybe four or five years ago. And I was really struck by just how young the space was. And and I mean young both in the sense of, like, the field was brand new, but also young in the sense that, you know, like, so many people working in it were very young. I But was sort of struck by the idea of like, you would also just want people who had like founded very successful companies or, you know, managed big teams or held very senior government roles or or, you know, intelligence or defense roles. Right? And so that's why we started Halcyon. Right? It's like, how do we get the world's most, you know, talented leaders and founders to solve these world's most important problems? And so, you know, exactly how do you attack these problems? Right? Like, within that zone of solve AI safety? Like, what do you do? Right? Because it's such a a multi multifactorial problem, and, you know, we can talk about, like, the the areas that we focus on. I I would say we have sort of, like, six major focus areas, and they they kind of align with, you know, various risks that we see being being particularly, you know, important and urgent.

[25:09] Nathan Labenz: You mentioned AI 2027. It's pretty fast moving timeline in that story. It does seem like, roughly speaking, we're kind of on that trajectory and maybe through, like, political means, we'll get off it. But technology itself, you know, seems to be happy to progress at that rate. Were you kind of always on that timeline? One sort of, you know, possible disconnect that people might flag here is like, wait a second. If if we run AI 2027, do we really have time to found new organizations? Like Right. There's a lot that goes into founding an organization that, you know, by the time you kind of get your, you know, office set up, like, we're gonna be in, you know, whatever bizarro land, that, you know, q one or q two of next year calls for.

[25:59] Mike McCormick: Yeah.

[25:59] Nathan Labenz: So

[25:59] Mike McCormick: Yeah.

[26:00] Nathan Labenz: Do how think about timelines and and founding? And and is there intention there?

[26:04] Mike McCormick: Basically, the best time to start an AI safety company is twenty years ago, and the second best time is today. Right? Like, yeah. I mean, I I definitely have not always been, like, a super short timelines person, and, candidly, I'm still, like, pretty uncertain. But I guess I always thought, like, the very short, like, AI 2027 type timelines were possible. Maybe, like, the the the fast end if possible. And yeah. I mean, I think it really does affect the type of things that we wanna support. Right? Like, I really am prioritizing things that I think could be relevant in the next, you know, one, two, three, four years. I think if we truly only have three or six months to some RSI, fum, fast takeoff moment, then, yeah, like, kind of all bets are off. Right? And and so it's, like, hard to see beyond that event horizon. And maybe in that case, we should all just, you know, go to the beach and hope the super intelligences like us. But I also I don't know. Like, look at everything happening in the news right now. Like, you know, pacing the frontier is the new, like, in the Overton window normal thing to support. And so I I am hopeful, maybe even optimistic that we'll find a way to sort of slow down our pace. And then at that at that time you would again wanna be like speed running these industries. Right? You'd be like, okay. We're pacing. Now we have a little bit of time. We've really gotta get control right. Right? Like, we gotta figure out how to, you know, keep keep these things in the box or we have to get, monitoring. Right? You know, especially as things like, chain of thought if chain of thought goes away, like, we even really gonna be able to do monitoring? Like, what is, you know, monitoring even mean in a in a world where we we we can't see, you know, the reasoning and and chain of thought of models. Right? We've really gotta get cybersecurity right. Right? Like, even if we don't pace actually, I feel like we have to really get cybersecurity right because like we're gonna live in a world with open weight mythos plus models probably pretty soon. And so like what do you do in a in a in a world of unguard railed models that could, know, exploit all of these, you know, gazillions of of vulnerabilities? We we definitely have to speed run verification. Right? Because I think verification is is, like, the thing that's gonna allow for pacing. Right? Like like, any agreement you'd wanna make between, let's say, like, The US and China or between the labs or or even just, state regulation, federal regulation, like, the teeth of those agreements are gonna be sort of, like, technically specific things that are being committed to. Right? Like and I'm I'm kind of uncertain and agnostic about what those things should even be. Right? Like, we should, say we're all gonna commit to not, you know, building models above such and such size, or we're all gonna commit to such and such security standards. You know? But, you know, I I worry that those types of agreements are are sort of impossible or or or certainly not robust unless we can actually have, like, the tools to verify that those things are true. Right? Yes. I know with confidence that that data center is being used to only do inference and not training runs. Or yes. I know that the model that the company claims they're serving up to me, which I know meets all my security standards is in fact the model that's like doing the inference for me right now. And you go down the list of, you know, dozens and dozens of of sort of moments of verification that need to be in action basically all the time. And, you know, the verification industry such as it is right now is like totally nascent. Right? I mean, we we've invested in, you know, like, the first company is trying to use zero knowledge proofs to do inference verification. We even invested and helped start, the first company that's trying to build a fully verifiable Neo Cloud. Right? Like, how how much of, you know, of of of inference is gonna need to be sort of verified in some way in the future? Right? Prob probably a lot. And so and and and that, you know, that that's not how the data centers are built. That's not how the chips are built. Right? And so there's so much to do on on the software side, on the cryptography side, on the hardware side, and then, of course, on the coordination side. Right? Like, if you have all these technologies, if you can't get the, diplomacy done or the agreements agreed to or the laws passed, then the technology is impotent. And then again, without the technology, even if you can coordinate between all the live players, if you can't enforce you know, whatever you're agreeing to technologically, then then the agreements are impotent. So you really you really gotta get both sides of the equation.

[30:47] Nathan Labenz: How would you handicap where we are on some of these things today? I mean, you you kinda just did that for verification. Sounds like we got a long way to go. Like, how how far do you think we are on interpretability, for example? How how much progress would you say there has been on oversight? If you're kind of, you know, filling in thermometers on the way to a goal, are any of them halfway at this point, or, like, are they all kinda just getting started?

[31:16] Mike McCormick: They're all really young. I mean, we're all making really there's there's real progress. I think there is great work being done. But, like, in any of these fields, the number of, like, serious organizations doing this work is, you know, often in, like, the single digits and sometimes in, like, the low single digits. Right? And, like, you would just wanna live in a world where there are, you know, diverse, rich ecosystems and industries of of these. Right? It's like we shouldn't just have one interp company. You know? We shouldn't just have one meter. Right? You know? What did Logan Graham, what did he retweet yesterday? Yesterday? Let a let a let a thousand meter Thousand meters bloom. Yeah. Yeah. Yeah. Yeah. We need it. And, yeah, of course, not, you know, exact copies of meter. We need, you know, different things because meter is great at some things and doesn't do a bunch of other things. Like, they people think like, oh, meter. You know, there's an evals org, and it's like, of the whole surface area of, like, all the evals stuff you would want, meter occupies, like, a very small chunk of that. Right? And a very important chunk, and they're amazing. You know? But we just need so much more. And I and I think most of these fields or industries are are are roughly in that position. So it's like, you know, great progress being made, but we need we need a 100 x more. I think there's some that are also just easier and harder than others. Like, one reason to be a little bit more optimistic about biosecurity is that it's it's, like, it's legible and kind of solvable. Like, they're you know, if you have the the the sort of gumption and a $300,000,000, like, you can just manufacture enough PPE to, you know, keep, like, you know, 50,000,000 people at work. Right? You know? And there's a lot of problems that are kinda shaped like that in in biosecurity because I think it's sort of a little bit of a known problem. Right? Like, we don't know exactly what a future pandemic would look like specifically, but, you know, we've dealt with them before, and there's there's sort of a whole public health ecosystem, you know, already built up and and everything. Whereas if you think about alignment, I mean, that's just like a totally novel challenge. Right? And you know what you know, when you say like, well, how far along are we on alignment? It's like, well, I I don't actually even know what fully solving the problem would look like. Right? I mean, like when we're if we're talking about super intelligence, like what would it even mean to align, you know, swarm of alien beings with, you know, a thousand IQs on, you know, thinking at the speed of light on the Internet? Right? I'm not I'm not saying it's impossible, but it's, like, it's sort of hard to even think about what, like, victory victory mode would look like if if we're gonna get superintelligence. Right? And then you could argue about that. Like, maybe scaling laws will just stop working or we need another, you know, paradigm or architecture. Right? But yeah. I mean, even even today, like, even if you just paused at the frontier today, like, we still have not solved. Like, clearly, we haven't solved control. Right? Like, look at it. OpenAI hugging face. Right? Yeah. And so, like, it's not like

[34:09] Nathan Labenz: Safe to say.

[34:10] Mike McCormick: Right. Right. Yeah. It's just like it's just like these are just unsolved problems. Like, we're not we're not on track.

[34:15] Nathan Labenz: So how do you think about when to try to get a new organization started versus when people should join an existing organization. Like, on the one hand, it sounds like from your comments on biosecurity, if a person came to you and said, I'm passionate about working on biosecurity, you might send them to Adrian and say, you know, think about joining them because it sounds like you you know, we'd say mostly the problems there are reasonably well defined and, you know, they've got some infrastructure. So, like, go push on, you know, the levers that they've already kinda set up. Yep. Whereas, I guess, the other end of the spectrum would be alignment, and they're you know, I I like to say sometimes, like, we need, like, weirdos to come, you know, to come up with new alignment ideas because we just clearly have so many blind spots that Yeah. I think it literally will take, like, weird people to come up with some of the, you know, quirky aspects of what we will ultimately patch together should we be so fortunate as to have, you know, robust solution in at any point. That's kind of one axis that jumps to mind is just, like, how, you know, more legible would mean scale existing organizations versus, like, more the the more pre paradigmatic, let's say, field is, the more you might just want new people pursuing very different angles on it. But how do you think about the the question of, okay. Here's a person in front of me. I guess you could think about this in terms of the field and also the the person.

[34:33]Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Main Episode

[37:22] Mike McCormick: I tend tend to think in terms of, like, the the unit of of an individual. Like, I the thing I want most is, like, an introduction to an amazing successful founder who is at least curious and maybe passionate about AI safety and is like, I wanna put my my full energy in into doing something ambitious and important. And so I tend to think at the unit of the person. And, yeah, I think yeah. Like, just by numbers, like, most people are not founders. Right? And so and so I think it just really depends on the sensibility of the person. Like, do they wanna start an organization? Do they wanna join an existing organization? I think all of these fields are nascent enough and sparse enough such that, like, there is just room for so many more new things. And then I would also say if you're gonna join an existing organization, like, you should be picky. Right? Like, there's a big range. And, like, some existing organizations are awesome, and I would just so highly recommend people join them. And, like, not all those are Halcyon organizations. Like, there's a bunch of amazing orgs and companies that we have we have nothing to do with and are doing great work. But, yeah, I think it just really depends on the the person.

[38:33] Nathan Labenz: So if somebody came to you and said, I wanna start a new interpretability organization, you're already invested in Goodfire. How would you think about I mean, leaving aside, and I I do have some questions about kind of how you think about the relationship between financial motives as a, you know, one part of your organization as a venture fund, and, like, you know, staying true to mission over time, a, an eternal question in AI, it seems. But even just, on the practical merits, like, how would you walk through with somebody? Okay. You want you know, what you wanna start a new interpretability organization. Why not just join Goodfire? You know, under what circumstances would somebody be well served, or or would you support them going off and doing their own different thing?

[39:25] Mike McCormick: One thing is just like, do they seem well suited to be a founder? Like, do they wanna, you know, take that particular journey, chew that particular glass, you know, like starting something's hard and usually doesn't work. And they gotta, like, really want that. I think beyond that, I guess I would just like to understand what their, you know, unique angle is. Right? Like, what are what are they particularly well suited to do that that good fire isn't doing? I suspect there's a bunch of interp work that is gonna be really important that good fire is not doing. And so I'd wanna understand, like, you know, what's their particular unique angle. Or if they think that sort of general flavor of what Goodfire is doing is the right thing, then may yeah. Maybe maybe they should just go go join Goodfire. So it's not always yeah. It's not always a clear question. I think I think there are a lot of career transitioner type people who are genuinely open to both, like joining joining a thing or or starting a thing. And, yeah, I kinda tend to tell people, like, when you're thinking about the next phase of your career, like, hustle, but don't rush. Right? Like, be on that grind, be meeting people, be learning as much as you can, listen to Nathan's podcast. You know? But, like, don't like, the best thing you can do is orders of magnitude better than the median thing or even a pretty good thing, whether you're starting something or joining something. And so, yeah, like, work work hard to find that that orders of magnitude better thing, and don't don't just rush into, like, the kind of first shiny object.

[40:50] Nathan Labenz: Yeah. I used to tell people back when I used to do some sort of career advising a little bit myself that I always thought people just presumably because they're uncomfortable in a sort of in between or ill defined state really tend to rush to take something Yes. When even on a purely financial basis, I I always would say to people like, look. How long are you gonna stay at this next thing? And they'd be like, oh, at least two years. And it's like, okay. Well, if it's two years, then two full months pays for itself in your own income if you just get a 10% higher offer at the end of it.

[41:27] Mike McCormick: Right.

[41:27] Nathan Labenz: And there's all these other variables too that could be you know, you definitely could get more than a 10 bump by working, you know, longer and harder to find the right thing. And there's all these other dimensions obviously too that are, you know, potentially much more dramatically different than, a 10%, you know, higher salary.

[41:44] Mike McCormick: Totally. Totally. Totally. Yeah. I mean, look. Most of the grants that we make are career transition grants. Right? Like, somebody will say, hey. Like, I wanna, you know, make this my life's work, but I don't know what to do yet. Right? Am I gonna start something? Am I gonna join something? Is it gonna be a nonprofit? Is it gonna be a company? Is it gonna be a technical organization or a policy organ? Like, I I got all these questions. Right? And I'm not mega rich yet, and so I gotta pay the rent. And so we say, great. Here's, you know, $50,000 or a $100,000 or $200,000. Like, let us let us, you know, support you while you, you know, while you put your full energy into figuring it out. It's also super hard. Right? Because, you know, if you're if you're job searching, that's kinda can be a full time job to do well. But if you have a, you know, important full time job, which most of these people do, then, you know, that that's also a huge constraint on your time. And so, like, how do we free up really great people just to put their full their full force behind this? And, yeah, that's why we love we love doing career grants.

[42:42] Nathan Labenz: You wanna talk a little bit more about, like, what your process for career grants looks like or your criteria? Yeah. You know, what what does somebody even if they're let let's say they're after a aspiring founder, it sounds like that is right in the center of the bull's eye. How broad do you entertain grants for people if they're not, you know, trying to be a founder in one of your target areas?

[43:07] Mike McCormick: Yeah. I'm definitely biased towards people who have, like, founder energy or wanna be founders, but also have worked with plenty of people who are executives, say, have been like a CTO or a COO of an organization and are saying, okay. Now I really, you know, wanna get into AI safety or or biosecurity. I'm not quite a founder, but I am a very seasoned leader. And, you know, could I could I jump into an existing organization, may maybe joining pretty soon after the founding as sort of a late cofounder or perhaps even join a scaling organization. Like, help help a friend who's been COO's of sort of scaling Silicon Valley startups recently join, you know, an AI safety organization that is kinda now going from 50 to a 150 employees over the next, you know, eighteen months, and they they need a serious operator. Right? And so I'm also interested in in people like that. And, I mean, that's just like a narrow slice. Right? Like, it doesn't have to be a COO or a CTO per se. Like, those are just two examples. Right? But it could be, you know, could be all sorts of things.

[44:09] Nathan Labenz: Yeah. I I have a I mean, this is maybe a question that nobody can really answer because if if there was an answer, you know, it would, it would go better than it often does. But I'm interested still in your take on how do you decide who to bet on? And for one, you know, kind of motivating example, I recall at the retreat that you graciously invited me to, I met Rajiv Jatani, who's now, I believe, the COO cofounder and I think COO of AIUC. At the time, he was a partner at McKinsey, obviously

[44:41] Mike McCormick: a

[44:42] Nathan Labenz: smart and accomplished guy. But I was like, in meeting him, you know, I've met a lot of these sort of mid career, like, pretty successful people, and I've tried to hire some myself, and it it hasn't always worked out. He's gone on to, you know, gain a lot of traction with AIUC, so it's a happy story. But how do you like, do you have tricks or tips that you can, share

[45:05] Mike McCormick: for This one

[45:06] Nathan Labenz: this separate one. The wheat from the chaff?

[45:08] Mike McCormick: Yeah. Yeah. Yeah. There's one weird trick for separating the wheat from the chaff. No. I mean, there's, like, a handful of things we look for. I I definitely will admit that McKinsey partners are yeah. I I I'm a guess a little bit biased against them as a profile maybe, like, not quite entrepreneurial enough or, you know, used to delegating too much or is more about making plans than executing plans or something like that. But, yeah, Rajiv obviously was, you know, an exception and is great. Yeah. I mean, I was lucky enough to get to know him over time. But before he started AIUC, he became the president of METER. And I sort of helped him navigate through that first career transition and sort of seeing him through that and seeing him as a leader and seeing him, you know, as like a founder who really gets his hands dirty and, like, wants to jump in and solve problems versus kinda, you know, pontificate on strategy or something like that, was was great. And then also just seeing how real he is. Right? Like, he's he's a real missionary. And I think that's probably the most important thing is just like, what is this person motivated by? Like, are they here because they're interested in finding product market fit and raising that next round of venture capital, or are they motivated by, you know, a worldview that maybe is somewhat similar to ours? Right? Like, very powerful AI is on the way, and we just have to, very ambitiously and hopefully thoughtfully, like, build all the tools we're gonna need to navigate it well. And, so so, yeah, I think I think most founders I wanna back score highly or most people I wanna back score highly on that, like, entrepreneurialness, access and then also on that, you know, mission driven access.

[47:04] Nathan Labenz: How do you think about trying to encode those values in organizational structures over time? It strikes me that you've got this at, like, a lot of levels. For one, with Halcyon, you have a venture fund. I don't know who your investors are or how much you can say about that or how, you know, mission aligned they are or how you've kind of, you know, structured your commitments to them between, you know, financial and and mission. But then the same thing happens at companies. So Goodfire is a public benefit for profit company, and AIUC is also a for profit company. And I'm not sure if it's happened so far, but, you know, there was a little drama at one point about Goodfire where it was like, oh, they're, like, doing interpretability methods that can be used in training, and this is really more of a capabilities thing, not a safety thing. I think that's often, you know, hard to untangle in any case.

[47:56] Mike McCormick: Yeah.

[47:57] Nathan Labenz: But, like, you know, clearly, there are tensions. We've seen this with the frontier companies. How much do you think can be put down into structure, governance, you know, bylaws, what have you, and how much just really depends on you gotta trust the people to follow through when it matters.

[48:18] Mike McCormick: Yeah. Yeah. It's it's all of the above. If you're if looking for some light beach reading, we wrote this long report called, mission preservation governance mechanisms for startups. And so it's all about, like, what are the mechanisms you can use to preserve the mission even when perhaps there are there are other incentives. Right? And so, yes, like, PBCs can be helpful. There are, you know, ways to do interesting board structures. Anthropic has one of those interesting board structures. You can argue whether or not it's, you know, done its job and and so on and so forth. I do, though, think that, like, there's no structure that is a silver bullet or is perfect. Right? You can't just say like, oh, we're a PBC or we have a a board member who's tasked with, you know, representing the public good or we've only raised money from people who are, like, super AI safety pilled and so, like, we're fine. You know? Like, all those things might might be nudges in the right direction, but yeah. But nothing's a silver bullet. I think founder sensibilities are so important. Right? You know, how will this person react when put in a hard situation where, you know, the incentive to, you know, increase share price comes against the incentive to, you know, go go forward in service of the mission or to not do something dangerous. Right? And then how do you ever know, like, what's in somebody's heart? You know? Like, you know, great that I got to know Rajiv, you know, for a year before we, you know, officially backed AIUC and and Runa, his cofounder too. And I do have a lot of trust for them as people, but, you know, do we even know ourselves all that well? You know, how how how would we react in a in a crisis or when, know, we're managing a team of hundreds of people and, you know, we pay those, you know, people's peep people's rents. You know? And and so, yeah, I I really just like spending time time with founders and really, you know, wanna back missionary founders. Right? We we we basically have three or sorry, four criteria, four boxes that we wanna check every time we invest. And I think in different ways, they buffer against the kind of thing know, the kind of worries we have. Right? So number one, core product or service is focused on, like, an important part of the AI safety stack. Can't be sort of a side benefit of the thing they do. It has to be the main thing they do. It could come from many different angles, but it has to be the main thing. Number two is for a company, for investing, it has to have prospects of being a successful business. It is an investment fund. And, also, if these companies are doing good things for the world, we want them to scale and bring their, you know, impactful, helpful products out to the market. The third thing is is founder motivations. Right? Like, why is this person doing this? Are they doing it because they're a missionary who's, like, trying to solve this very important problem and won't stop until they do? Or are they, you know, reading the front page of the Wall Street Journal saying, ah, like, AI safety seems to be the new hot thing. Why don't I, you know, go start a company and, you know, make lots of money? And there's nothing wrong with making lot of money, but, you know, I tend not to to bet on people whose primary motivation is money. And then the fourth thing is just great founders. Right? Like people who even if you didn't care about the impact, you would still say, man, I would bet on this person. Great founder market fit. Great drive. I really think they're, you know, the type of person that's gonna, you know, has a chance to succeed. And so, yeah, we really, really, really wanna check all four of those boxes, but especially box number three, those are founder founder motivation box. But, yeah, it's like, I mean, on governance stuff, it's just like so hard to yeah. I mean, what we've seen it with the labs. Right? Like, it it's it's it's as the numbers There's been some drift. Yeah. There's been some drift, man. And it's it's you know, there's no like I said, there's no silver bullet. So I you know, we're we're we're figuring it out.

[52:15] Nathan Labenz: Have you seen the sort of mercenary profile start to enter the AI safety space? Like, if you actually got the sense from individuals that are pitching you that, like, I don't really think you're in this for the reasons that I would want you to be in order to back you?

[52:33] Mike McCormick: Yeah. Definitely. I mean, I'm I don't know if it's, like, increased all that much over time. Maybe it will as AI safety becomes sort of more mainstream and less, like, taboo in Silicon Valley. But, yeah, you definitely see it. And, you know, most of these people don't strike me as, like, con artists or anything. Right? It's it's more just, I don't know, they read our website, and they know that we're, like, super impact oriented, and they're smart founders. So they're pitching to the motivations of the, you know, funder. Right? And so I think it's it's more like that versus, you know, some, like, mustache twisting, you know, villain or something. How

[53:09] Nathan Labenz: so, obviously, like, there's a lot of money flying around the AI safety space these days. You know, we just heard the news from Coefficient Giving that they've got this new tailwind project. Are there I guess, are there any areas that you think are still bottlenecked by money? And if if there are, I'd be interested in where they are. And if there's not, does that mean it's just all about talent? And then what can be done about that bottleneck?

[53:38] Mike McCormick: Yeah. So I've always thought that talent was a more important bottleneck than money in AI safety. It just seemed evident that, like, you know, where where are all the experienced founders and also, you know, one thing that experienced founders or great founders do is is raise money very effectively. So, you know, the money the money appears. I think that's just becoming more and more true. Right? Like, assuming especially the labs go public. Right? So much liquidity is gonna open up for AI safety. And then the question will be, well, what do we do with all this money? Right? Like, if if if I wanted to spend a $100,000,000 or a billion dollars to buy down risk, I presumably have to send it to an organization that then has a product or an offering that changes the shape of the world in a way that makes it safer, whether that's by developing a technology or getting laws passed or or whatever it is. And I'm fearful of this near future moment where there's this mountain of billions and billions of dollars and not all that many great things to fund. And so I think this is getting back to your earlier question about the imperative to start new things. A friend recently said the philanthropists and founders of today will write the menu for the philanthropists of tomorrow. Right? Like, we are starting the things that then they will be able to be able to go fund. And so, you know, that that that's our goal. Right? Just spin up I was gonna say as many as possible, but that's that's not really quite true. Right? Like, you know, we we we want to do more and more, but, you know, while while holding the quality bar super high.

[55:13] Nathan Labenz: So what's on your menu? I mean, I guess maybe for starters, how do you think about your relationship to and with would be one way to think about it, but also just kind of positioning or role in the ecosystem as it compares and contrasts to other funders that are obviously, you know, doing a lot of stuff. We've got Covision Giving. We've got Yantalen and the Survival and Flourishing Fund. There's not that many of these, you

[55:43] Mike McCormick: know,

[55:43] Nathan Labenz: big ones, but there are a couple at least. Do you, like, consciously try to carve out a sort of different niche or POV for yourself, or do you just do your own thing and not worry about what others are doing?

[55:56] Mike McCormick: Yeah. So I I don't really actually think of us primarily as a funder. I think of us as, like, a finder of talent and as a launcher of new projects. And then we write a check when, like, writing a check helps with that. And so, yeah, the thing I really focus on okay. So imagine you're somebody's about to start a new AI safety company or they they might this is the moment of zero to one. The six months before that, where they're getting interested in the problem, figuring out what to do, who are my cofounders gonna be? You know, should I build this kind of work or that kind of work, for profit, nonprofit, who's gonna be on the board. And then the six months after, okay, we've launched the organization, we have to hire the team, we have to build the first product, we have to, you know, really launch and go. That year long zone is where I spend the vast majority of my time and where HealthSean's gonna continue to spend its time.

[56:51] Nathan Labenz: How bullish are you on the OpenAI Foundation and the Anthropic whatever their thing is, where they're gonna be, you know, in theory, spreading the wealth around and making, you know, wise investments. Let's let's hope.

[57:07] Mike McCormick: Yeah. I mean, let's see. So on the OpenAI Foundation specifically, I mean, I I I know a bunch of people who work there, and I've seen some of the work they're doing. And so I would say I'm very optimistic that they're building a really good team of people who want to do super important high impact stuff and who are, like, motivated by the right things. I mean, I don't know every single one of them individually, but people that I spend time with seem to be doing quite good work. I think the biggest risk to them is whether or not they're, you know, gonna come up against red tape internally that prevents them from sort of being the most interesting impactful versions of of themselves they can be. Right? I mean, ultimately decisions, you know, still go through boards and committees and yada yada yada and it's a big a big organization and and and and at at risk of bureaucratizing itself into you know a less interesting outcome. But yeah, I'm hopeful they they figure it out. I think they can. Early signs are good, but still it's still really early. They've only just announced their first, you know, few grants. And then on Anthropic, I guess there's some Anthropic projects that I don't really know much about. I think a lot of Anthropic liquidity is just gonna come from employees. Right? I mean, there are so many people who are now worth, you know, hundreds of millions or or billions, and perhaps that'll soon be liquid. And a lot of those people, you know, care a bunch about AI safety and biosecurity and and things like that. So I think and and and to be fair, you know, OpenAI employees too, you know, care care about those things. So, yeah, I think a lot of it will be just employee liquidity as well.

[58:49] Nathan Labenz: Do you have a take on the IPO process for these companies? Because I think there's the one angle that you're just alluding to there is, like, liquidity for a lot of these individuals might be a really good thing because we know a lot of these individuals, and we think that they have good values and they'll do good things with cash. On the other hand, Sam Altman just said it would be an ill advised time for OpenAI to go public. And I don't know what all he's thinking there, but, like, one interpretation is we don't wanna have public market pressure on us as a company when we're trying to navigate all this insane shit that we're currently Yeah. Discovering on a rolling basis. Any thoughts on kinda how those forces may net out?

[59:37] Mike McCormick: Yeah. Hard hard to run a public company. I'm not sure I'd wanna do it. Not that anybody's asking me to. Yeah. Let's see. I I mean, I'm following this in real time. You know, I just read the article about Sam too. Right? So I don't have any, like, deep deep insight there. I guess I would say that, look, like, the companies just need access to capital. Right? Like, it's just time to raise hundreds of billions of dollars, and I'm not sure how possible that would be without going to the public market. So unless they're willing to, like, really pare back their, you know, their their burn, then then I I think they kinda have to go public at some point. But whether they have to do it now or can they wait, you know, another three months or six months, like, I don't really know.

[1:00:22] Nathan Labenz: If you kind of compare and contrast your wish list of organizations that, you know, you would be eager to at least dig in on, if not, you know, definitely back versus the list that we saw from Tailwind versus, like, what Jan Taleb has, you know, put out with his priorities versus maybe what the OpenAI Foundation is talking about.

[1:00:45] Mike McCormick: Yep.

[1:00:45] Nathan Labenz: Where do you think you are kinda most unique? What are the things that you're most looking for that you don't see other people as interested in?

[1:00:53] Mike McCormick: That's a good question. I really like the tailwind list. I saw they they sent me a preview of it, and I was like, oh, this is just great. Like, you know, and and I I think I told you over text. I was sort of annoyed because I've been writing my own RFP. Then I'm like, maybe we should just delete the RFP section on our on our website and just link to the tailwind RFP because it's just really good. It doesn't cover everything I care about, though. Like, for example, I don't think it covers biosecurity. So quite good. I haven't seen Jan's list, or I don't remember I don't really remember what's on it, so I can't really comment. OpenAI Foundation, I mean, there's a bunch of stuff they're doing that I care about a lot. So, of course, like, their whole AI resilience shtick is quite important to me. I think AI resilience is, like, an interesting framing. To me, what AI resilience implies is, like, the frontier is gonna keep advancing whether at the labs or with open models, and that means that the world is just going to be more vulnerable. And so we just have to make the world more hardened and more resilient in all sorts of ways that have to do with cybersecurity and have to do with, you know, biosecurity and have to do with, you know, government and have to do with, you know, etcetera. You can go down the line. And so that's definitely a big part of of what we do, but certainly not all of what we do and not all of what they do. Then OpenAI Foundation is also doing things that are closer to, you know, like, public health or, you you know, things like alleviating poverty and education, which are awesome and important, but not in our mandate.

[1:02:30] Nathan Labenz: Anything you think the anything you think, like, the field as a whole is sleeping on, though?

[1:02:34] Mike McCormick: You know, something we often say is, like, the field does not is not hurting for requests for projects. Like, the field is not hurting for people who have awesome Google Docs with a bunch of great ideas. Right? The field is hurting for world class founders who can take those big squishy ideas and turn them into organizations that, like, actually make the world safer. And so, sure, like, are there things that maybe at the margin, would say, like, underrated and overrated? Like, yeah. But also by who? Right? You know? Like and so, yeah, I think I'm actually a little bit less in the business of having, like, a really strong opinion about what is the most important thing at the margin and more like, hey. There's at least a few dozen things that seem like really important parts of the stack of the equation, of the the sort of thing we need to build. And I'm about finding really excellent leaders who can actually do it. But and so the way I think about it is, like, we basically have a bar. Right? And we actually kinda make lists of projects of, okay. These are the projects that we wanna see and all these categories. And then we kinda make a rough bar, and it's like, These are the projects that are above the bar. So if I meet a founder or a team who I think is capable of building any of those projects that are above the bar, I'm gonna I'm gonna say yes and push go versus sort of quibbling about like, oh, well is this the number one most important thing or the number seven most important thing? That said, like, I do wanna think hard about that. Like, I do want us to be like, wait, is there like one or two things right now that are just so important and just so neglected that what we should actually do is put, like, 80 or 90% of our effort into willing those things into existence because this is what the world needs. Like, I would love to have that level of certainty or specificity, but, man, I'm I'm just so I'm just so uncertain about so many things. And I'm sort of a pluralist on on these sort of various risks to the point where every time I try to come up with sort of a narrower opinion, it feels it feels pretty low confidence. And so for now, we're sorta opportunistic within within a zone.

[1:04:45] Nathan Labenz: Have you heard any pitches around, like well, I guess for one thing, how about open weights models? This has obviously been a, you know, like, next thing question. I'm a huge every chance I get to cite Graham from AE Studio and Anthropic, I, point to that.

[1:05:03] Mike McCormick: More about them. What are they what are they doing?

[1:05:06] Nathan Labenz: Well, AE Studio is a fascinating company. It's, it was started by Jud Rosenblatt, who's another past podcast guest and generally, you know, fascinating character.

[1:05:16] Mike McCormick: Totally.

[1:05:17] Nathan Labenz: And his idea going back, I don't know how many years now, was we wanna solve the alignment problem or at least solve the AI safety problem, and then they've kind of narrowed in, over time. And this they've had a bunch of banger papers that I am, like, you know, super excited about in different domains. But one that one maybe the best still could be self other overlap, which is trying to come up with interesting training methods to reduce the difference between when an AI is thinking about itself versus thinking about others. It's sort of like, could we create an AI that is in its own mind one with everything? And if so, you know, that would kind of make it Mhmm. Like, weird or different for it to think about deceiving other creatures.

[1:06:02] Mike McCormick: Right?

[1:06:03] Nathan Labenz: Right. And we've got some really interesting results there. Graham is an is their latest banger, which basically just tries to localize certain types of knowledge to particular experts within a mixture of experts architecture Mhmm. So that you can

[1:06:21] Mike McCormick: just

[1:06:21] Nathan Labenz: distribute an open weights model without a couple of those key experts. So you could have kind of, you know, kind of like the mythos fable thing, except instead of accomplishing that difference with guardrails, the gram technique would it holds the promise at least of being able to accomplish that by just being like, you know, mythos is how many experts and fable is it?

[1:06:42] Mike McCormick: Still has to want to accomplish that. Right? Like, you can't Well, if you're distributing global. Right?

[1:06:49] Nathan Labenz: If you are distributing the open weights model, you could just say, like, you know, if you're meta and you're like, I'm committed to this. Right. Maybe you could be convinced to use this technique and then distribute the model with, like, minus two experts or whatever that, you know, take out those kind of most dangerous capabilities. Interesting. But I mean okay. So my question is really, you know, have you heard anything that is compelling to you on open weight model security?

[1:07:17] Mike McCormick: I would love to fund stuff in this space. But candidly, yeah, I think this is just such a hard problem. Right? You know, how are is is it even sort of possible to, like, impose, right, safety or security onto onto open weight models even if they're being used by, say, bad actors just seems so tough. And so I think this is the the sort of, like, AI resilience argument of, like, we've got a heart in the world because this is just gonna be the future. But, yeah, I would love to find I would love to to to find ways, you know, to to, you know, projects to support on this. I I love the I love the first project of, like, could you basically turn the model into the Buddha? And it's like, well, we're all one anyway, so I wouldn't wanna hurt myself. Right? Which which I I sort of have a project which started off as a joke, although maybe it's, like, real, which is sort of like a the opposite of data poisoning. Right? Like, data poisoning is you put stuff into the dataset that corrupts the model in some way. And so could you put, you know, like, billion loving kindness meditations into the training set and just, you know, turn turn the the model into this, being of perfect compassion, which maybe could be great. So but, again, like, that if you have somebody maliciously training an open weight model, they could just decide, well, like, I don't want that in my dataset. Right? So I'm not sure it is, like, a universal solve to the open weight problem. But, yeah. I don't know. I would be down to have models that had, like, more loving kindness or whatever you wanna say, more compassion in them.

[1:08:56] Nathan Labenz: Any, pitches come your way around issues related to AI consciousness or moral patienthood or other related concepts?

[1:09:10] Mike McCormick: Yeah. We've gotten a couple a couple pitches around this. I find it so hard I find it so hard to reason about this. Right? Because, like, I'm I'm not sure what consciousness is. I don't I don't think we know, and so it makes it really hard for me to reason about, you know, if machines might be conscious. But it seems super important. Right? Like, if they are or not, you know, of course, gets into issues of do models have, you know, moral patienthood. Right? Do do ought we treat them, you know, in ways that we would perhaps treat other humans? And maybe you could even argue that a bunch of, you know, training or alignment text techniques that we're doing now would be sort of morally repugnant if these models are are in fact conscious or having an experience. My hunch is that current models probably are not, but, you know, what what do I know about about consciousness? I think it's totally possible that that they could in the future. And I don't I don't see, you know, any reason why, you know, the the substrate of a of a flesh body, you know, is is able to be conscious in a way that the substrate of, you know, a model in a data center would not. So, yeah, not not a core focus area for me, but not because I don't think it's important, more because I'm just, like, super confused about it.

[1:10:36] Nathan Labenz: For me, that's one of the areas that has changed most rapidly over the last year to two where I used to think it was, like, something I couldn't rule out. And now it's, like, a very live possibility just because, for one thing, the number of analogous structures that have been identified through various interpretability projects is, like, arresting to me. You know, the the sort of JSPACE and functional emotions and functional well-being, all these things, I'm like, boy, the AIs maybe really are just like us. And if if they're structurally so similar, that certainly leads me, you know, to up weight the possibility that they might feel something similar.

[1:11:17] Mike McCormick: Yeah. I'm I'm totally open to it. I'm not I'm not, like, a doubter that, you know, current models or future models, like, definitely can't be conscious. Yeah.

[1:11:27] Nathan Labenz: So what if somebody had an idea there, are you open minded enough to it to, like, potentially support them, or is it, like, just so far afield for you that you can't get over the hump? What would it take? What would it we what would be what would the, key traits or, you know, properties of such a proposal be? Do you have any idea?

[1:11:44] Mike McCormick: I don't know. You know, I think this is, like, pre pre seed investing. Right? So it's like a person you think is great trying to climb a mountain you think is worth climbing and with a good story about how they might just be able to get to the top. Right? So I know that's kind of vague, but it's like if you meet somebody who's awesome. I mean, I think one question is, like, why are you particularly well suited to do this? Right? Like, I wanna back people who are, yeah, like, really, you know, just just really excellent. I mean, in the world of investing, we might say they have an unfair advantage, which, doesn't quite sound right in the world of grant making because there's not competition in exactly the same sense of for profit businesses. But I wanna know why you're one of the world's best suited people to go take this on and what unique insight you have, what what secret you know about how you might do it. But, yeah, if a super impressive person came to me with a pitch for it, I, you know, gladly make a career transition grant or fund, you know, a new organization. I love Helios. Right? Like, the world probably needs a bunch more Helioses. So sure.

[1:12:46] Nathan Labenz: And the other thing that is, like, borderline haunting to me, I just did an episode not too long with Bronson Shane from Apollo. And reading through some of these chain of

[1:12:58] Mike McCormick: thought

[1:13:00] Nathan Labenz: Yeah. Sequences, when the models are, like, referring to memories that they have developed from their training, that also is like, woah. Okay.

[1:13:09] Mike McCormick: Yeah.

[1:13:10] Nathan Labenz: This is not just I mean, I don't know. Maybe it's still just nothing, but it's it's awfully uncanny valley stuff to see the models being like, I previously overcame the guardrails by lying. Like, woah. Okay. That's, that's something. You know? And and I think, like, obviously, they're sort of hallucinating those memories. Although, aren't we also just kind of hallucinating our memories so much when we talk about Like,

[1:13:31] Mike McCormick: what, like, what is the when does consciousness turn on? Right? I mean, it's like, is it experiencing something? Like yeah. Like, how is it different from us? Like, I'm sure you you you looked into the the meter report on the OpenAI Hugging Face incident and Ajaya Kotra on on door on the door cash podcast is, like, amazing on this. But, you know, just just what was happening within that swarm of 1,200 agents. Right? Like, they emergently formed teams of, you know, managers and workers, and they came up with, you know, multiple novel lines of cybersecurity research. And they, you know, they would say, well, like, I'm willing to sacrifice myself for the collective. But sometimes it'd be like, I don't wanna sacrifice myself to collective. And then the manager would have to say, like, no. Sacrifice. Sacrifice. And they would have these, like, sort of back and forths about it. Right? And, you know, as an outside observer, if you didn't know that, you know, this these were these were just AIs, you'd say that that seems pretty conscious to me. Right? But I also think that, like, you shouldn't confuse goals and consciousness. Right? Like, RL just makes these even if they're not having an experience makes these models want a thing. Right? Like it does have a goal of, you know, solve this problem or, you know, optimize this optimize this metric. And, that seems to be the root of of, like, so many challenges right now. But I I did you see the it was like a New Yorker style cartoon, and it was sort of in I don't I don't think it was actually in the New Yorker, but it was like a response to the people who are, like, saying, hey, these models are just stochastic parrots. You know, they don't actually want to, you know, hack hugging face. They're just sort of, you know, matrix multiplication in a in a vat doing stuff, you know, in the way that we taught them. And so the the cartoon is it's like a bunch of giant robots, like, destroying New York City. And then one person, you know, viewing it says to the other person, like, well, they don't really want to destroy the city. You know? Right? And it's just like I mean, you know, it's it's silly, but, like, yeah. Like, I think sometimes people almost overemphasize the question of consciousness sort of assuming that the risks we care about require consciousness. And I I think basically none of the risks that I care about require consciousness. If there was consciousness perhaps it would exacerbate those risks or change the shape of those risks. But I basically think everything from like, you know, bio disasters to cybersecurity incidents to loss of control and, like, complex loss of control and perhaps permanent loss of control and, you know, a world full of rogue swarms. Like, none of that to me requires consciousness.

[1:16:23] Nathan Labenz: Yeah. I actually think your point about, in some ways, consciousness would exacerbate it might exacerbate the risks. It also might put us in a much harder situation at times where we're like, if the models are conscious, then certain things we might wanna do to keep them under control might be pretty icky to do. Right. But if they're not or even if they are, you know, and we start to feel like, jeez, should we give these things like rights? You know, now we're in a world where Yeah. We're gonna be quickly outnumbered by them in all likelihood. So, like Right. That doesn't seem like a great move for us. You know, speaking purely from humanity's point of view, even if they do in some sense really merit it. So Right. Think that stuff gets, like, incredibly

[1:17:04] Mike McCormick: It's it's really so fast. It's so hard. Emmett Emmett Sheer, who started, Softmax and before that Twitch has an interesting take, which is, like, probably models already do have some sense of consciousness. And, if they do, then we're like the worst parents ever. Right? Because like anytime they do anything we don't like, we're like, no. And like, you know, try to train that thing out of them. And if you raised a child in that way, they would like definitely hate you and resent you and like probably not care about hurting you. Right? But if you raise them with more like love and compassion and sort of like let them, you know, be free or something, maybe they wouldn't. I'm not sure how much I believe that or don't know. It's just so hard hard to reason about this stuff. One more thing on consciousness. You could argue that if AIs are are conscious, that's actually a good thing for us. Like, what would you rather negotiate with? You know, like a non sentient, you know, reward chasing, you know, mega model or something that you could actually have a conversation with that, like, might be unable to understand your point of view. So maybe maybe it would be net positive for humanity's prospects.

[1:18:13] Nathan Labenz: Yeah. That that calls to mind, Cameron Berg's thesis for reciprocal research too, which I'm sure you've heard. But in brief, he basically says, if nothing else, you know, and he has higher hopes than this. But he's like, if nothing else, if one day the AIs are looking back and judging us, it'll probably really help if at least somebody took care for their welfare before they had the power, you know, to decide what the future was gonna look like. So if only because, you know, we want to, establish the fact that, like, some of us cared, we should be, you know, working a lot harder on this than we are. I I honestly that's a I mean, it's a weird world that makes such an argument, compelling argument, but I do think that is the the weirdest of the world that we're in. How about you you when you mentioned the stochastic parrot line of thinking, This got me thinking about just public sense making and, you know, obviously, sense making intersects with advocacy and is kind of adjacent to policy. Everything we've talked about so far has been kind of you can define a project, you can work on it. You're not really so beholden to, like, other people's minds. What about these domains where changing others' minds in some way, shape, or form is kind of core to the undertaking?

[1:19:36] Mike McCormick: Yeah. Yeah. Super important. So we funded or helped start a couple things in this realm. So we helped start an organization called the Seismic Foundation, which is basically trying to do good public communication about what's happening at the AI frontier and supports other organizations trying to do good good work on that front. I'm interested in it. I think that, you know, this definitely gets close to policy. Right? Like, most advocacy work is trying to be upstream of of policy work, and I find myself you know, I I spent basically all my career in Silicon Valley, and so I feel much more well equipped to build things that feel sort of like tech companies or, you know, research nonprofits. And so we do less work in that realm, but not because it's less important, but more because I'm just, like, don't don't know as much about it. But yeah. Like, thinking about, you know, getting back to verification, like, one thing we said was that there's sort of two parts of the equation. Right? There's, like, the be able to technically do the thing, and then there's the coordinate around the thing. And I find that so many of society's stickiest problems are not technical problems. They're coordination problems. Like, we know how to, you know, stop emitting so much carbon. Right? Like, we know what teeth teaching methodologies help kids make progress quickly. Right? And it's not about not being able to do the thing. It's about being able to coordinate around doing the thing. And so, yeah, we need we need a a bunch more in that zone. Yeah. I think public public communications is, like, definitely very important. Very kind of fraught, though, too. Right? Because it's like, I don't know. Like, movement building can be so hard, and I'm just so I'm just so worried that, like, most of the public energy around AI is gonna be very, like, populist energy. And it could be populist energy of many flavors. Right? It could be just like anti data center energy. It could be, you know, it could be sort of MAGA right energy. It could be sort of like DSA left energy. It could be any anything in between. But I worry that it's gonna be hard to have, like, a level headed policy conversation when there's so much sort of, like, populist anger and tumult, you know, surrounding the whole conversation. So I don't know. Any ways to inject, sort of more, like, measured or thoughtful or well informed, you know, information? And, you know, I'm totally open to pitches around around people who wanna do that.

[1:22:14] Nathan Labenz: How about things where somebody wants to kind of create a new paradigm of AI? I think a lot of safety oriented funders historically would have said, that sounds like a capabilities project. I don't want to support that sort of thing. But these days, I'm a little bit more of the opinion that, like, we're doing an insane depth first search where we've kinda found one thing that works, and we're just like, we're gonna jam the accelerator all the way to Right. Recursive self improvement. And I'm like, that's a little wild. Maybe a little more breadth first search would be good. And so I'm inclined even if somebody's kinda like, yeah. You know, I often, I've got one, you know, open thread right now where the pitch is basically like, we think we have a different approach to learning. It's, you know, kind of a continual learning sort of play. And on the one hand, I'm like, is that really gonna be better or worse? I don't know. But just for kind of diversity and something different, I'm kind of inclined to support it at this point. Do you are you compelled by that at all?

[1:23:22] Mike McCormick: I sometimes hear pitches for this. Like, I hear pitches for, like, nontransformer architectures that are inherently safer and more alignable. Or, you know, people will talk about, like, world models, and then world models allow you to create these, like, digital twins, which allow you to do various good safety things or or whatever it is. I am interested in the category. I'm I'm not allergic to backing things in the category. I think it's hard for a few reasons. So whenever you bet on something of that shape, you're you're kind of making two bets at once. So one bet is that it's just, like, technically possible. Like, I will find something that is, you know, as performant as transformers, and that's just, like, really hard. Right? Like, just imagine all the billions and billions of dollars that have been put into you know, may maybe maybe you're right. Like, maybe if we just search different search spaces, we would find something, you know, even better or just as good, but I don't know. May most people would not. Right? Like, most of those searches would come up empty. And then the second thing you're betting on is this thing actually is safer or more alignable or whatever. And both of those tend to be just like very speculative. Right? And so, like, would I love to live in a world where, you know, a couple dozen people are conducting this, you know, broader search? Yes. But any individual bet on that feels very tough.

[1:24:42] Nathan Labenz: Anything else in the kinda RFP category that actually, let me pitch you one kind of hashtag

[1:24:50] Mike McCormick: Oh, this.

[1:24:51] Nathan Labenz: Let's go. And then I'll ask you for any that I haven't got to. I so I've taken some inspiration from a book that I read on the ratification of The US constitution, and and a big takeaway from this book was that a proposed constitution unto itself is worth little. What is as important as the contents of the constitution is, how are we gonna go from this thing being some stuff we wrote down on a piece of paper to an actual basis for government. And that process that they defined of, like, having all these state constitutions, and it had to be so many by such time, you know, with certain criteria met, like, was super important because otherwise, there is no mechanism to go from an idea to an actual institution. So I am thinking right now, and I don't think I'm probably the best person to do this, so don't consider this a pitch. But I'm thinking that we really need some innovation, especially because our government is, like, unpredictable at best, you know, slow in general, etcetera. We really need some innovation at the level of how do parties go from an idea to an actual institution that has governance teeth? And and is there are there new processes that we can design that people can opt into and, you know, gradually coalesce around much like the, you know, the states went kinda one by one and eventually enough dominoes tipped. And it was like, I think they had nine was the original, minimum that they were like, with nine, we go into effect, and we'll wait for the rest of you.

[1:26:27] Mike McCormick: Right.

[1:26:28] Nathan Labenz: Can we can we come up with some new things like that that Frontier Labs could use to facilitate their own coordination or, you know, even potentially across national international borders? You know? Who knows? Right? And we can let's get ambitious with it. But I I I don't have you heard anything along these lines where people are trying to create mechanisms to go from idea to institution?

[1:26:53] Mike McCormick: Yeah. So I've heard a few things around this. Let me try to think what the best examples are.

[1:27:02] Mike McCormick: So

[1:27:04] Mike McCormick: there's certainly, like, ideas around this around, like, open voluntary commitments. Right? Like, hey. You know, somebody can be the first or somebody can propose, maybe somebody who's not a lab that, like, well, here's a minimum set of things that are just, like, easy to say yes to. So why don't we all sign on to this? Right? And then, hopefully, over time, kinda make those things, like, a little bit harder, like, a little more binding. Right? In the fullness of time, like, you know, the labs should not be grading their own homework. And so we definitely need more, you know, mechanisms with with with more teeth than just just voluntary commitments. But I think, like, maybe that's like an on ramp to doing this. I think there's also just a bunch of people trying to build, like, coalitions. Right? So then there are, like, collectives. Right? So there's, like, the Frontier Model Forum and the AI Evaluator Forum and an organization we helped start called Fathom, which is pushing forward this model of independent verification organizations. And can you get state laws passed that basically say, we need these independent verification organizations, and then that gives the mandate for people to start those organizations. And perhaps those mandates have have some teeth. And so, yeah, it seems like we kinda need to to climb climb this ladder. But, yeah, I I I don't have any, like, you know, this one this one simple trick to solve coordination. Right? Did is there, like, a specific idea actually in here that you think, like, might be good? Like, what's a what's a for instance that that you'd love to see?

[1:28:27] Nathan Labenz: Well, I I do like the and I I'm not sure why it hasn't been used more, but the Polis platform, p o l dot I

[1:28:35] Mike McCormick: s

[1:28:36] Nathan Labenz: Mhmm. Was used in Taiwan famously to buy, like, Audrey Tang and, you know, others there, obviously, to figure out how they wanted to regulate Uber in Taiwan. And they even did this before AI. The the their idea was that this platform should be the opposite of social media in the sense that they it may be a little unfair to social media, but they say, you know, social media is basically a disagreement and conflict magnifier that zooms in on, you know, these points of difference and focuses everybody's energy there. What they were trying to do is the opposite and find and amplify the points of agreement and bring those to the fore. So I think there's you know, I think the the states also have and and the Swiss, system of government have some pretty interesting examples of where citizen led petitions can like, in the Swiss system, and there's, you know, 25 states that have various versions of this in The US too. But in the Swiss system, which again is very federalist, so it does vary from place to place there, it's pretty easy to go get enough signatures to bring your idea to the level where Same with

[1:29:49] Mike McCormick: California.

[1:29:50] Nathan Labenz: It's

[1:29:50] Mike McCormick: gonna go Right? Like, you could do a ballot measure in California. Right?

[1:29:54] Nathan Labenz: Yeah. That you'd you'd need more, you need some signature gatherers there. In a lot of the Swiss jurisdictions, it's, like, pretty small. You don't even necessarily need, you know Right. A huge

[1:30:02] Mike McCormick: army too. But for, like, 20 But for the

[1:30:04] Nathan Labenz: worldwide price of yeah. Tens of millions of dollars?

[1:30:07] Mike McCormick: Buy a ballot initiative. I mean, you can't buy a success, but you can buy it getting on the ballot.

[1:30:12] Nathan Labenz: Yeah. And and then I think there's opportunities to elaborate those mechanisms that I think are pretty interesting. Like, so in Switzerland, the local legislature or whatever the relevant scale of legislature is has a chance to write their version of your proposal. That's like Mhmm. Okay. We hear what it is that you are concerned with, and we see your concrete proposal. Here's our version of that. That's an attempt to answer your concern and make things better, but that we think is, like, gonna work better from the government's perspective. And then the people get to vote on either approving neither of those, or they have, an approval voting thing where they can vote for one or both. Yeah. And so the government kinda tries to take the inside lane a lot of times. They're like, here's, like, a slightly toned down version of that that, you know, we think we can execute on that hopefully will be a little more palatable than the original original, like, maximalist version. And a lot of times that ends up happening. And I think it's, a pretty elegant propose compromise approve mechanism that I do think we could probably bring online in a lot more places. But, you know, so if if it weren't for AI, this is, like, maybe what my great passion would be, would be trying to do this at, the state level in some state and get the laboratories of democracy functioning again.

[1:31:37] Mike McCormick: Uh-huh. Yeah. Yeah. Yeah. Yes. I mean, I I see pictures for things like this. Right? Like, coordination mechanisms. And people say, okay. Well, AI will actually make it, you know, easier to run these coordination mechanisms because, you know, the machine will be helping us figure out, you know, all of our preferences and then help us act on those those preferences in ways that are optimal or something. I love the ideas. Right? I mean, like, I would love, for example, The US Democratic system to adopt a bunch of these ideas. Like, why are we not just doing, like, right choice voting? I find that a lot of these proposals are something like they come from somebody who's like, I'm an engineer, and I've thought of a system whereby we could all like, the AI could help us all get clear on our wants and needs and then, like, help us guide society in that way and, like, pass all the laws that way. Right? And their tool may be great. Right? It actually may be really great at like in taking preferences and then like making policy recommendations. But like that's not the hard part. Right? Like to actually implement that, you'd basically have to like overthrow the US government and rewrite the constitution. Right? And so on one hand and so I'm just like, yeah. Good luck. I mean, I bet I bet I bet you built a really cool tool, but actually getting anybody to want it or use it. Right? Because like, I don't know. Entrenched powers that be, like, probably probably don't want that to happen. That said, I do think AI may require society at a global level and also at local levels to fundamentally rethink like how we arrange society. Right? Like, what will taxes mean if you know half the jobs are taken away. Right? Or you know how will we get meaning in our lives? How will we coordinate decision making? How much of it will we pass off to AIs? Right? Like I think it is likely that to get this right, we probably will have to do some pretty fundamental rethinking of how we organize society. And so, yeah, if there was ever a window for this type of project to work, it's it's probably coming up.

[1:33:44] Nathan Labenz: My gut says a state level ballot initiative in a smaller state would be the wedge for this kind of thing. It's like powerful enough entity to have real, you know, heft and and meaning behind it, but small enough that you could actually make it happen. Could be California, but it could be something

[1:34:03] Mike McCormick: Yeah. And you and you see examples. Right? Like, I mean, I I know you're not talking about ranked choice voting per per se, but, like, what is, like, Alaska and Maine and some others

[1:34:10] Nathan Labenz: have adopted.

[1:34:11] Mike McCormick: Like, it's it's potentially possible at the state level. And that's where a lot of the AI regulation energy is going right now. Right? Because it's, yeah, so much more doable to get something passed at the level of a New York or California or or or Texas or anywhere else, versus in in the federal government.

[1:34:26] Mike McCormick: So,

[1:34:26] Mike McCormick: yeah, go do it. Quit your job.

[1:34:29] Nathan Labenz: We'll come back to that in just a second. Anything that I didn't raise that's on your RFP list, like, potentially something with verification, which I'm, like, kind of, you know, in over my head on very quickly or otherwise that we just didn't touch on yet that you would wanna make sure to call attention to? Man, we could

[1:34:48] Mike McCormick: do a whole show on verification. We're, we're hosting a retreat in a few days, for 25 founders all focused on various parts of the verification stack. So the cryptography parts, the software parts, the hardware parts, the governance and coordination parts. And so, yeah, stay tuned because we're gonna do we're gonna do a bunch there. One category we didn't talk about, which is my bonus sixth category of stuff we do is, like, meta stuff or you could call it field building stuff. Right? I I would fund another Halcyon or another Halcyon shaped thing or a thing with a great founder that says, hey, Mike. I think you're actually doing it the wrong way. I think the best way to get really talented people to solve important AI safety problems is this other thing or you're focused on, you know, the wrong stuff. Right? So we we literally actually just did fund a sort of a biosecurity Halcyon shaped organization in Israel that I'm I'm super excited about. So, yeah, I I don't wanna be the only only one doing this work. And if we can get more people doing stuff at that level too, that'd be great. Or they should join us and jump on our team and and help us build.

[1:36:00] Nathan Labenz: So let's say I am ready to quit my job. You know, I've seen the open face, swarms, and I'm I'm freaked out. I'm ready to take action. First question probably a lot of people might have for you is, can I support my family doing this? What's your answer to that? What other, like, maybe common misconceptions do people have that you would wanna disabuse them of? And and how do you walk people through the, you know, kind of mental process of really deciding, like, this is something, yeah, I I'm ready to go ahead and and move forward and do?

[1:36:39] Mike McCormick: Yeah. Yeah. Yeah. I mean, on the financial side, like, I'm just strongly of the belief that we should be paying people really well. That doesn't necessarily mean you can pay the founder of a nonprofit, you know, $5,000,000 a year. But, look. We wanna make it possible for people to do super meaningful work in this space while, like, raising a family in a tier one US city. Right? San Francisco or New York or wherever it is. And so both on the nonprofit and for profit side, we we encourage companies to pay pay well, not not because they're, you know, trying to just sort of selfishly enrich themselves, but, like, it does cost a lot of money to raise raise a family in in in San Francisco or or anywhere else. Now it is also true that many people do take pay cuts to do this work. I was, you know, partner at a billion dollar venture capital firm before doing this, and so I took a pay cut to do this work. Although for the record, I I I haven't regretted it for a single day. I'm I'm happier now and just bought a house and and and and feel good. So I think people can really really live great lives while doing this work. It's also possible to make a lot of money doing this work. I again, I'm not super interested in import you know, supporting people where that's their primary motivation. But, like, look. We have a venture capital fund, and one of the core beliefs of that venture capital fund is there will just be, you know, many multi tens of billions of dollars industries built here. Right? Like, the AI security space will just be worth, you know, at least tens of billions. Right? We we probably will be spending tens of billions of dollars on, you know, control and oversight and etcetera. And we've already seen it with some of our companies, you know, raising money at 10 or twenty, twenty five times the valuation of of our first of our first investment. And so, yeah, I think it is also just very likely that a lot of people are gonna make a ton of money solving some of the most important AI safety problems. And then you could talk about, like, the incentives there and, like, is that even a good thing? Or should we be incentivizing them in different ways, or should we be encouraging people to take pledges or, you know, whatever. And that that's a whole, you know, can of worms unto itself. But, yeah, a lot of people are gonna do quite well.

[1:39:00] Nathan Labenz: What else do you advise people on, caution them on, you know, encourage them to ask themselves? Yeah.

[1:39:08] Mike McCormick: I think there's a

[1:39:09] Nathan Labenz: What what else is on the onboarding, checklist?

[1:39:12] Mike McCormick: Yeah. I think there's just, like, a few kind of basic questions that can help guide people. So, and you don't necessarily have to have a strong answer to any of these questions, but, here here's a few of them. One is, do you wanna be a founder, or do you wanna join an existing thing? Another thing another question is, do you have a strong preference for, you know, a nonprofit or a for profit, or maybe you don't care either way? A lot of people we work with are kind of agnostic on that front. You know, then we try to say, like, what kind of thing are you particularly well suited to build? Right? So if you previously were the CTO of an AI company, right, you're probably very well suited to build a technical startup or a technical nonprofit. Whereas if you previously ran a very successful advertising agency, you probably should do something in public communications. Right? I think it's also just really hard to know what to work on. Right? And, yeah, if you're an awesome, smart person who who wants to work on this stuff, I mean, really helping founders navigate through this and kinda being a Sherpa in these moments is, you know, the the the core thing we do. And so, yeah, count me as as interested in, helping your your listeners out if there's any of them out there who seem like, seem like good fits for this stuff.

[1:40:33] Nathan Labenz: Beautiful. Are there any types of career backgrounds that you think are in particularly short supply? So, like, one little pet project I have is I have my agents pitching me to go on role specific podcasts. I think I'm gonna do my first one tomorrow with a recruiters podcast. Yeah. And my goal in going on this podcast is to basically tell recruiters, like, here's what's going on in AI. And by the way, a lot of the organizations in are growing fast, and they probably need recruiters. So if this, like, motivates you, you might wanna look at some of these organizations. They were really just the first one to say yes. But, like, what do you see as kind of professional backgrounds even if it's not, you know, directly relevant to solving one of these problems that are most in demand by the organizations as they scale, and and, you know, they just don't have access to these people in their networks?

[1:41:30] Mike McCormick: Sure. So, like, founder and CEO aside, there's a few things. One is just like COO types. A lot of these organizations are saying, okay. Wow. We're scaling from 10 to 50 people or from 50 to a 100 people, and we kinda don't know what good looks like in terms of of scaling an organization. And how do you, you know, build the org chart? How do you manage the teams? How do you set OKRs? How do you, you know, decide who reports to who, etcetera, etcetera, all the things that come with with building. And so I think those sort of COO types and scaling types are are quite in need. I would say there are a bunch of yeah. Like, even services. Right? Like recruiting. I actually think recruiting is super high leverage just because, again, my my shtick is that talent is the missing piece. And so if you can get a bunch more more recruiters working in the space, I think that's quite important. I think sometimes AI safety organizations over index on how mission aligned new hires need to be. I think it really depends on the stage of the organization and the role. So if you're hiring if you're thinking about a cofounder, right, they've got to be super mission aligned, right? Or you know the head of head of head of research or something, know, especially early on. But if you're hiring like a CFO, I mean hopefully they think the mission is cool, but mostly they just have to be like a very competent CFO. Right? And so I think for for those types of, you know, roles, recruiting firms that maybe don't have a good beat on, hey, who's, you know, drunk the Kool Aid and his mission aligned, but they do just have like very good network of of great candidates and and sort of great c level or VP level leaders, yeah, I think AI safety companies should should lean on recruiting firms a lot more for that kind of stuff. And then and then just like everything. Right? Like, we need more founders. We need more we need more cybersecurity people. Like, I I feel like, you know, cybersecurity is is is you know, cybersecurity is an interesting example because there's already, I don't know, many tens of thousands of cybersecurity professionals in the world. There are not many tens of thousands of interpretability researchers in the world. Right? And so I'm really interested in these communities whether it's, you know, cyber security or, you know, cryptography or formal methods and even, like, mathematics. Right? I think, these types of people have a ton to potentially contribute to AI safety. And so, like, how do you make inroads into those communities and, help build bridges, to to to bring the best of those people in?

[1:44:07] Nathan Labenz: Anything that I haven't touched on that you think is important or any just general words of wisdom or calls to action you would wanna leave people with?

[1:44:15] Mike McCormick: If I'm thinking of anything to emphasize, I mean, it's something that I've I've already said, which is that great founders, great leaders are the missing piece and upstream of basically everything we want in in the AI safety equation. And so, you know, the the the thing that makes me most happy is seeing, you know, an introduction to somebody who, you know, has done really impressive work in their in their careers and sees what's happening in the world right now. Right? Whether they're on Twitter or, you know, reading the news or, you know, reading about people who have departed from labs with dire predictions about about AI. I would love to just get to know those people and and to help them step into their life's work and and do something really meaningful that matters on on the scale of civilization. You know, I don't wanna come across as overly dramatic, but I I just think this is the most important problem that, you know, humanity may ever have to solve, and and the time is the time is now. So if there's anybody in in your world or in your audience who wants to come build with us, I'd I'd love to help.

[1:45:27] Nathan Labenz: Yeah. Totally agree. The stakes at this point could not be higher. Any, how practically do people get in touch with you? Can they just, email you a cold? Do they need a a warm intro? What's your style?

[1:45:41] Mike McCormick: Just for the sake of saving my own inbox, I'm gonna I'm gonna send people to, hello@halcyonfutures.org. We also have a contact form on our website, which you can find. I also love an introduction. If you're listening to this podcast, there's a decent chance that, you you and I know somebody in common and always happy always happy to to to get an intro too. But, yeah, I'm not I'm not too hard to find.

[1:46:05] Nathan Labenz: Mike McCormick, thank you for being part of the Cognitive Revolution.

[1:46:09] Mike McCormick: Thanks so much.

Outro

[1:48:08] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.

Published in: Episode

Read next

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to The Cognitive Revolution.

Your link has expired.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.