Latest news
Episode

Write, Change, Recall, Forget: MongoDB's Pete Johnson on How Retrieval Drives Agent Performance

MongoDB Field CTO Pete Johnson traces the company's shift from lexical to vector and hybrid search, new $rankFusion and $rerank query stages, and the Voyage AI acquisition's contextualized chunking and embedding techniques.

Nathan Labenz
Write, Change, Recall, Forget: MongoDB's Pete Johnson on How Retrieval Drives Agent Performance
67 min read

Watch Episode Here

Listen to Episode Here

Show Notes

ℹ️
The following notes are AI-generated, based on the episode transcript. Please listen to the episode for the full conversation.

Nathan's guest this episode is Pete Johnson, Field CTO of AI at MongoDB, and the conversation is really two conversations woven together: a history of database architecture, and a status report on the still-unsolved problem of agent memory. Pete opens with a framing device that recurs throughout — he was born in February 1970, four months before E.F. Codd's original relational-model paper that gave rise to SQL. The relational model, he explains, was built for a world where storage was the scarce resource, so normalization — splitting data across linked tables to avoid duplication — was the rational design choice. MongoDB, whose first commit landed in October 2007, was built for a different scarce resource: time. Cloud and mobile applications couldn't tolerate weekend downtime or departmental usage patterns, so MongoDB stores data as JSON (on disk, as a binary form called BSON), denormalized into documents rather than split across tables — trading some disk efficiency for far fewer round trips per query. Pete is careful to correct a common misconception here: MongoDB isn't "schemaless," it's schema-flexible — documents in the same collection can have different shapes, which is what gives it room to evolve without the cascading migration pain of a rigid SQL schema.

From there the conversation moves into MongoDB's build-out of search and retrieval, which Pete narrates as an unusual, backwards-feeling sequence. It started in 2020 with lexical search: MongoDB noticed customers standing up their own Apache Lucene servers next to their MongoDB clusters just to get keyword search, so MongoDB built that capability directly into Atlas. Vector search followed a few years later — technically straightforward for MongoDB, Pete argues, because a vector is just an array of floats, which slots naturally into the flexible-document model as one more attribute. That set up hybrid search: combining lexical search, vector search, and metadata pre-filtering in a single query. The plumbing behind that combination has itself gotten simpler over time — new aggregation-pipeline stages called $rankFusion and $scoreFusion let developers combine lexical and vector results (by rank or by score) in one API call instead of writing that merge logic themselves, and a companion $rerank stage does the same for reranking. Pete frames these as MongoDB applying something like "bitter lesson engineering" to its own product — removing artisanal glue code as the underlying capability matures.

The 2025 acquisition of embedding-model maker Voyage AI (reportedly around $220 million, per Bloomberg reporting on the deal MongoDB officially announced) gets a substantial chunk of the conversation, prompted partly by Nathan's observation that the price is under 1% of MongoDB's market cap — what should that imply about the relative defensibility of infrastructure versus models? Pete's answer leans on ease-of-use rather than raw model capability: Voyage's team, led by Stanford professor Tengyu Ma (now MongoDB's Chief AI Scientist), built features that reduce the iteration burden on developers. "Contextualized chunking" — most recently shipped as voyage-context-4 — tries to escape the classic chunk-size tradeoff (small chunks lose context, large chunks dilute retrieval fidelity and cost more to store) by embedding a chunk's local text alongside document-level context in one pass, rather than forcing developers to hand-tune chunk size through trial and error. Voyage's models also use Matryoshka Representation Learning — the "Russian nesting doll" technique that orders embedding dimensions so you can truncate a high-dimensional vector down to a smaller one without re-embedding your whole corpus. And per Pete, embedding quality itself is not commoditized: Voyage models sit near the top of the Retrieval Embedding Benchmark (RTEB) on Hugging Face, with a reported ~14% quality edge over some competing embedding models — notable partly because Anthropic doesn't ship its own embedding model and recommends Voyage instead.

The back half of the conversation turns to agentic memory, which Pete calls "the hardest problem with agents right now." His potted history: early-2023-era ChatGPT had no memory and no live data access; 2025 brought tools and MCP to give models fresh, external information; and the newest problem is that looping agents accumulate context faster than a fixed window can hold usefully. The naive fix — cramming everything from a session, or from the last several days of sessions, into the context window — creates what Pete calls "token maxing," both a cost problem and, he argues, a quality problem, since research suggests the first and last several thousand tokens of a context window matter most and the middle can just add noise. He points to Uber's much-discussed 2026 token-budget overrun (Pete's framing was "in 13 weeks"; reporting generally describes roughly four months) as the cautionary headline behind the industry's pivot away from stuffing context windows and toward more selective retrieval — including a newer pattern he calls "taxonomic memory," where an agent pulls only the handful of domain-specific terms relevant to a given loop rather than an entire glossary. The hardest sub-problem within all of this, Pete says, quoting a colleague, is captured in four words: "write, change, recall, forget" — and forgetting, specifically knowing when something has gone stale, is the piece nobody has solved. He cites ElevenLabs — reportedly running on the order of 40 million agents in production, by his account — as MongoDB's most sophisticated memory customer, though he notes they haven't been public about their specific architecture.

Pete's practical advice for enterprises threads through a recurring "3-box" framework from his own writing (see "There's No SKU for AI"): pick problems where you already have good data and, critically, already have metrics, so you can attribute an ROI number to whatever AI changes. Call centers are popular first targets precisely because cost-per-call and volume are already tracked. He's skeptical of lines-of-code as a productivity metric for AI coding tools like Claude Code or Google Antigravity, preferring idea-to-production-deployment speed instead — a theme he's also written about at length. And in a closing detour prompted by Nathan's question about global AI adoption, Pete pushes back on the assumption that the US is furthest ahead: after visiting seven countries this year, he says the two most sophisticated enterprise AI deployments he saw were in Mexico City and São Paulo, attributing it to how thoroughly cloud/AI infrastructure has now spread globally compared to earlier technology waves, where hyperscaler data-center access was a real geographic bottleneck.

Topics Covered

  • SQL vs. NoSQL, schema vs. schema-flexible — the history from 1970 to 2007 and why the tradeoffs have changed
  • MongoDB's build-out: lexical search (2020) → vector search (2023) → hybrid search with pre-filtering → the Voyage AI acquisition (2025)
  • $rankFusion, $scoreFusion, and $rerank — collapsing hybrid search and reranking into single API calls
  • Chunking strategy and "contextualized chunking" as an attempt to escape the size/cost/quality tradeoff
  • Matryoshka embeddings and truncatable dimensionality
  • When "it depends" isn't good enough: speed, scale (~100K vectors), and retrieval-quality thresholds
  • Embedding-model quality as a non-commoditized differentiator; Anthropic's recommendation of Voyage
  • Bitter-lesson engineering applied to RAG pipelines
  • Agentic memory: short-term vs. long-term, taxonomic memory, token budgets, and "write, change, recall, forget"
  • Build vs. buy, the 3-box framework, and picking the right AI problem
  • Employee-facing vs. customer-facing AI deployment risk
  • MongoDB's MCP server and Agent Skills
  • The Voyage AI acquisition and what it implies about model vs. infrastructure defensibility
  • Data that's never been in a database before — video, audio, unstructured documents — and multimodal embeddings
  • Global AI adoption patterns outside the US

Resources

Quotes Worth Pulling

  • "It's not that we're schemaless, it's that we're schema-flexible."
  • "There's two kinds of developers. There's those that love MongoDB, and there's ones that haven't tried us yet."
  • "Most people think embedding models are commoditized, and that is not true. There is a very big difference that you can get in retrieval quality based on what embedding model you choose. Anthropic does not have an embedding model in market. They recommend us."
  • "We've been building databases for 60 years. We've been building agents for, like, 18 months, man. Like, nobody knows all the answers yet... Write, change, recall, forget."
  • "Things like bad data quality and bad security posture don't get solved by AI. They get amplified by AI."
  • "There's a presumption in other countries that the US is ahead in doing things that other people are not, and I found the opposite to be true... the two coolest things I've seen this year were in Mexico City and in São Paulo."

Sponsors:

Mercury: Mercury is the banking platform loved by 300,000+ entrepreneurs, with virtual cards and Spend controls for granular budgets, receipts, and low-risk AI agent purchases. Learn more and apply in minutes at https://mercury.com

Granola: Granola is an AI-powered notepad that securely transcribes meetings and turns rough notes into clean, structured action items. Try it free at https://granola.ai/tcr

Diffusion: Diffusion helps organizations build custom AI software factories that scale business outcomes, not just outputs. Cognitive Revolution listeners get a 25% service credit on their first engagement at https://diffusion.io/tcr

Deepgram Flux TTS: Deepgram Flux TTS brings lifelike AI voices with real personalities that handle interruptions, pauses, and natural conversation. Try all the voices free through September 12 at https://deepgram.com/keep-talking

Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

CHAPTERS:

(00:00) About the Episode

(03:15) Sponsor: Mercury

(04:56) SQL versus NoSQL

(11:24) Enterprise database choices (Part 1)

(18:30) Sponsors: Granola | Diffusion

(21:27) Enterprise database choices (Part 2)

(21:27) Schema flexible search

(33:50) Contextualized chunking tradeoffs (Part 1)

(35:12) Sponsors: Deepgram Flux TTS | Claude

(37:17) Contextualized chunking tradeoffs (Part 2)

(46:22) Retrieval quality thresholds

(53:17) Agent memory systems

(01:05:51) Enterprise AI deployment

(01:16:32) Voyage acquisition strategy

(01:23:38) Global AI adoption

(01:30:56) Episode Outro

(01:34:41) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution!

Today my guest is Pete Johnson, Field CTO of AI at MongoDB.

We start with a brief history of database technology, going back to the 1970 paper that gave us the relational model and, with it, SQL — unpacking how the relative scarcity of disk space led to the canonization of database normalization as a design principle, and how decades of Moore's Law, and the resulting evolution of constraints, ultimately led to the creation of MongoDB in 2007.

This also puts today's acceleration into stark relief, because, as Pete points out, AI pattern cycles are coming fast and furious. We've gone from being effectively forced to implement RAG pipelines by very limited context windows, to "RAG is dead" when we first got million-token context windows, to the brief token-maxxing window — which collapsed as fast as it arrived, with Uber, as Pete tells it, burning its whole 2026 token budget in 13 weeks — to today, where RAG is a priority again, now that data is increasingly available for AI systems to use, usage is scaling, and naively maxing out the context window costs multiple dollars each time.

Pete's key point, above all, is that agent performance — and especially cost-adjusted agent performance — depends heavily on effective retrieval.

With this in mind, MongoDB has continued to ship the improvements developers need to take full advantage of AI: vector search, rank fusion and score fusion, $rerank, and their own embedding models, powered by the acquisition of Voyage AI, which have cool features like a shared embedding space and a Matryoshka structure.

We talk about how developers are using these tools to build memory systems, the "write, change, recall, forget" loop he sees as an emerging pattern, and why "forgetting is the hardest part." Pete is refreshingly candid that nobody has this figured out — as he puts it, we've been building databases for 60 years, and agents for about 18 months.

He's also blunt about where enterprise AI goes wrong, and it's rarely the model: bad data quality and bad security posture don't get solved by AI, they get amplified by it.

And based on his many conversations with enterprises all over the world — he's visited India, Brazil, Mexico, and Europe already this year — we get a couple tips on how he thinks enterprises should approach build vs buy, and the surprising observation that the most advanced companies he's spoken to this year were in fact outside the United States.

With that, I hope you enjoy this speed run through database history and survey of the still-fast-evolving frontier of agent memory, with Pete Johnson, Field CTO of AI at MongoDB.

[03:15]Mercury: Mercury is the banking platform loved by 300,000+ entrepreneurs, with virtual cards and Spend controls for granular budgets, receipts, and low-risk AI agent purchases. Learn more and apply in minutes at https://mercury.com

Main Episode

[04:56] Nathan Labenz: Pete Johnson, field CTO of AI at MongoDB. Welcome to the cognitive revolution.

[05:03] Pete Johnson: Thanks so much for having me, Nathan. I appreciate it. I'm excited for this conversation.

[05:07] Nathan Labenz: I've been a

[05:08] Pete Johnson: database

[05:09] Nathan Labenz: guy in the past, and now we're all kind of everything guys these days with our agents greasing the wheels for us. But the I think one of the big themes that we're gonna get into is the quality of data is still critically important to making things work, and I'm excited to learn everything I can from you over the course of the next hour or so. Maybe just for quick starters, I remember first encountering MongoDB, like, fifteen years ago maybe when a young Guillermo Rausch, who's now the CEO of Vercel, was showing it off in the context of what at that time was, like, seemingly crazy futuristic Node. Js applications. Like, wow, you can make JavaScript do these kinds of things, and there's a wait. There's a database that works in this different way where I'm not constrained by the schema that I already defined. This was mind blowing stuff. I'm not sure we've got a lot of technical people in the audience that say AI engineer is kind of our number one profile, but I'm not sure everybody knows kind of the broad lay of the land when it comes to database technology. I thought maybe it would be helpful just to first contextualize SQL versus NoSQL, schema versus schema less. Give us the high level view so we know where Mongo sits in this obviously bigger database space.

[06:32] Pete Johnson: Sure. I like to think of this in terms of the history, in terms of the different kinds of problems we've had to try to solve with software over the last sixty years or so. And where it really starts for me is in June 1970 is when EF Cod, who's an IBM researcher, wrote the original white paper that gave birth to SQL. So it was June 1970. I happened to have been born in February 1970. So I am as old as SQL, Nathan. If you think about the kinds of applications we were building at the time as an industry and the kinds of hardware we had at our disposal at the time, the applications were very departmental in their nature. Right? Not every not every company had a computer and those that did, the people that were using it were often on the same floor or certainly in the same building. They were working nine to five hours. It was okay to have downtime on the weekend. And if you think about the big three piece of hardware at the time, memory, compute, and storage, storage was by far the most expensive of those three things in 1970. So if you think about how you might organize data, keeping in mind that the scarce resource was storage, you would of course do it in such a way that's now known as normalization. So you may have in university or in coding school, you may have learned things like first and third normal form, all about trying to lay out data on disk in the most efficient way so that you aren't storing any piece of data more than once. The simplest example I can give with that might be my wife and I share an address and maybe with a retailer might wanna know what our address is. There's two people at that address. You might have in one table the list of people. In another table, you might have the addresses. You might have a third table that links the people to the address. So these are only storing that address one time. And that's the most disk efficient way to do that with, like I said, some kind of normalization. Well, fast forward to 2007, which in October 2007 is when the first commit was made to MongoDB and the world is a very different place in 2007. The world has Internet. The world has Cloud. The world has mobile devices. IPhone came out in The US for the first time that year. So it's a very different world and the scarce resource in that world because we had forty seven years of Moore's law between those two dates. The scarce resource in that ecosystem became time Because the kinds of applications that we would write that were in the cloud or that we would write for mobile devices, you could not have downtime on the weekends. It was not people who were confined to the same building. It was not nine to five use cases. So it was very different world that required a very different kind of database, and that's what gave rise to NoSQL. Now our flavor of NoSQL, which like I said, that first commit was in October 2007, uses it as its core JSON. So you've mentioned the stuff with JavaScript. That's why that was so popular fifteen years ago or so is because what you get back from your API call is JSON. We're storing it in transit as JSON. We're storing it on disk, and a binary form of JSON we call BSON and that's part of why we're so fast is because we're keeping that format the whole time so that when you serialize it off a disk, when it goes through our servers, when it ends up in your client, it's essentially staying in the same format that whole time. So if you think about the kinds of applications that we've been building since 2007, like I said, disk space is no longer the scarce resource. If you take that address example, if you denormalize that data just a little bit, if you instead of storing it across three tables, if you'd have a JSON structure that has my name and my address and then a second one that's got my wife's name and the same address. You're duplicating the address, but what do you get in return? What you get in return is one disk read. So that's why it's faster. So instead of having to do three disk reads to get that same data, you can now do it in one disk read. Does that mean that every problem is solved by a no SQL database or by MongoDB in particular? No. But it also doesn't mean that every problem has to be solved with SQL. And that's how that sort of evolved out of SQL was invented at a time where the problems were slightly different and a lot of those problems still exist. SQL is certainly a valid solution to lots of different problems that we still have. But when you need speed, when you need scale, and increasingly, when you need better retrieval quality, that's really where MongoDB shines.

[11:25] Nathan Labenz: Reminded of the old saying that the problem has faded, but the solution persists. And Yes. There's yeah. I see that in a lot of parts of life these days. Tell me a little bit more. You're you were just kinda getting to it, but tell me a little bit more about I mean, Mongo is a pretty big company, $30,000,000,000 market cap. Obviously, has made serious inroads into some of the biggest companies in the world as customers. Give me kind of the next level of detail in terms of before the AI moment, then obviously we'll get into that in-depth. What are the kind of big decision factors that would push an enterprise to go one way or the other with their database decision? How are they do they typically do it on like an application by application basis these days, or do big companies feel that they need to have a single kind of consistent database technology for everything they're doing?

[12:24] Pete Johnson: Sure. It it's a mix. Like you just said, we're in, like, seventy five percent of the Fortune 500. But if you look at the overall market, you mentioned our market cap. We're about 2,500,000,000 is what we ended our revenue that was our revenue number for our fiscal twenty six. We're currently in the middle of our fiscal twenty seven, and that's about 3% of the 100 to $110,000,000,000 database market. Where most enterprises choose us is those first two things that I mentioned, when you need it fast or when you need it at higher scale. Because you're not storing disk, you're not storing that data across multiple places of disk, that's how you get the speed, that's how you can get better scale if you're if you're storing if you're gonna retrieve JSON because every modern programming language can serialize JSON into some object format so that you could use it immediately in your business logic. And that's the other aspect of time. It's not just wall time for the transaction, but it's also the developer time of what do I have to do with this wad that I get back from my API call and then make some kind of useful business decision out of it. But what we see is right solution for the right problem. And increasingly, what AI has done is it's opened up the number of problems that were a better foot fit for than a traditional SQL approach, especially those at higher scale. And that's why I mentioned the three things, speed, the scale, the retrieval quality has really become an important factor. It continues to especially later here in 2026 as you start to see people strategize around token maxing to evolve different kinds of agentic architectures in ways that we didn't see in the first half of twenty twenty six. Retrieval quality has become more important for AI, but it it it does end up being a mix. It depends on the corporation. Some of them will take it workload by workload, and for this workload, we'll use Mongo, and for a different workload, we'll use SQL. Others are taking an all in strategy. It really varies based on, in some cases, how much greenfield versus brownfield do you have in your portfolio. In other cases, it's where's my talents already. I was just at for a major automotive company here in North America. I was just at a hackathon

[14:32] Pete Johnson: where

[14:33] Pete Johnson: we trained up, like, 250 engineers and turned them loose for three days on a hackathon and they've got some new product ideas that fell out of that. So there there's there's no single right answer. Right? There's lots of ways to sort of walk through that forest. But at the end of the day, it does come down to right problem for a resolution for the right problem. And where we typically help people is with those modern applications that need that speed at scale and those gigantic use cases that increasingly need that better retrieval quality. I'm really interested though in

[15:05] Nathan Labenz: big three tech modernization, and I've a bit of a brush with it from time to time just enough to know that there's a lot of appetite, but there's also a lot of friction. What was your experience of and I'm sure a lot of these app concepts that people were working on are AI apps. Right? Without naming names, what more can you tell us about the kind of Gonzo on the ground experience in the big three? Some of it was customer experience kind

[15:32] Pete Johnson: of applications. Others was how can we make the dealers more efficient? How can we make the mechanics more efficient? Some of it was how can we make the in car experience more efficient? It really spanned the gambit of different types of use cases that a big three automaker might have. But really, it's I mean, I'll go as far as to say, like, our education system has a bias, a historical bias that thou shalt always normalize. And for a lot of use cases, that's valid, but there are an increasing number of use cases where these big three that I just mentioned matter. And increasingly, that retrieval quality one is becoming a much bigger deal for agentic architectures as you start to try to take better advantage of the context window than just stuffing it with a million tokens every time. And we can get into some of the details as to why we're a good fit for that and why that's emerged, like, in the wake of what happened to Uber earlier in the year with them burning through their token budget for twenty twenty six and thirteen weeks. That grabbed a lot of headlines, and there's been similar stories about where, like, maybe token maxing wasn't such a great idea to begin with.

[16:38] Nathan Labenz: Yeah. It was a phase. We've all we've been we've all been through our phases. I was really attracted to the schema less database model way back when because I'm just a free flowing person who loves to just make a mess now and tell myself I'll clean it up later. This is one of the things that I'm really interested in in terms of how DB interacts with AI applications because it seems like on the one hand, it's more tempting than ever to do that. Right? It's more valuable than ever to just be like, let me just dump all the data in here and like maybe an AI will make use of it sometime later. But then you do have interesting challenges downstream of that. One with your example of, like, the two records with the same address. Now you gotta update it in two places, so that's something to keep track of, obviously. I'm also kind of struck that, like, the advantage of latency you may have a different take on this, but it feels to me like the advantage of latency is somewhat reduced because now, like, everything gets put through a slow step anyway. And so the ability to, like, shave a little time off the DB call when I'm then gonna have to wait for tokens maybe matters less. And then I'm also thinking, like, if I do start just dropping all my data payloads into a sort of extra field, Now I'm gonna be doing, like, exactly what you're saying I shouldn't be doing in terms of just bringing a million tokens into context all the time. So take that in whatever direction you will. What do you see as kind of the biggest trends in application development that have bearing on your database choices as we are AI ifying everything?

[18:30]Diffusion: Diffusion helps organizations build custom AI software factories that scale business outcomes, not just outputs. Cognitive Revolution listeners get a 25% service credit on their first engagement at https://diffusion.io/tcr

[19:57]Granola: Granola is an AI-powered notepad that securely transcribes meetings and turns rough notes into clean, structured action items. Try it free at https://granola.ai/tcr

Main Episode

[21:27] Pete Johnson: Sure. So let me so first, let me push back on one thing you just said. Most people think that we're schemaless, and that's not actually true. It's not that we're schemaless, it's that we're schema flexible, That you can change the schema more easily over time because more of the data is denormalized and therefore centralized in a way. It's not a strict requirement unless you specifically say so that every document in a collection, so we don't have tables and rows. Right? We have a think of a JSON WAD as a document and you have multiple JSON WADs put together into a collection. Not every document in a collection necessarily has to have the same shape. You can have different shaped documents in the same collection. So because the schemas are flexible and because you can have different doc different document shapes in the same collection, that's why people assume that we're schemeless. But what it does is it gives us the ability to be malleable in a way that a traditional SQL schema is less malleable. I won't say not malleable at all because it depends on how you're laying out the data. But anybody who's ever had to change a SQL schema that's already in production and the cascading effect that has on multiple tables knows the pain of what I'm talking about that if you had some of that data denormalized, it's far easier to add attributes to a document that's already there and do so selectively in a way that isn't possible nearly the same extent in the SQL world. And this is where how we've implemented like, the history of how we implemented vector search, the impact that has on application architectures for agents ends up like, the details matter here. So if you'll allow me, I'd like to talk a little bit about that history. Does that sound okay?

[23:19] Nathan Labenz: Yeah. Please.

[23:21] Pete Johnson: Okay. So for us, it really started within an unusual place, and that is with lexical search. So in 2020, we noticed that a common use case for MongoDB was for people to stand up their own Apache Lucene servers, co located with wherever their their MongoDB clusters might be. The reason they were doing is they wanted to be able to point that Lucene cluster to different text fields in a document and be able to do keyword retrieval off of them. Perfectly reasonable things. We thought, well, as you might know, there's three different versions of MongoDB. There's community, which is you're responsible for the support and you're responsible for the operations. There's enterprise advanced, which most customers are using on prem, where you're responsible for the operations, but we're responsible for the support. Then there's Atlas, which is our managed service version of it, which you can deploy your instances on any cloud hyperscaler data center you'd like amongst Amazon, Google, and Azure. And in that form factor, we will do the support and we'll do the operations. So those are the different choices you have when you deploy MongoDB. And regardless of which one you choose, like I said, we noticed people standing up Lucene servers so that they could get keyword retrieval on what they have. So in 2020, we introduced what's now known as Atlas Search, which is on the managed version the managed version of MongoDB, you automatically just you just get it as part of that instance that that there's a lower priced tier where you just point it to attributes some text attributes that you already have in your data will automatically index them. You can do keyword retrieval search on those. Then the next logical thing was a couple years later, well, if you're gonna have lexical search, you might as well also have vector search. So think about this. What's a vector? At the end of the day, a vector is an array of floats. Right? You take some piece of data and whether that piece of data is text or an image or audio or video, you pass it to an embedding model of your choosing and what you get back is an array of floats. So to us, an array of floats is just an additional attribute to a document you already have. So for us to implement vector search was just add an additional attribute to the flexible schemas that we already have and now build a vector index on top of that array of floats, and that's your vector search.

[26:15] Pete Johnson: And what it created was this notion of being able to do these powerful hybrid searches as well. So if you think about the sort of the trivial example that most people learn MongoDB off of is suppose you have a book and a book might have a text field for the title. It might have a text field that is URL that points to the cover of that book. You might have an integer that's the number of pages, you might have an integer that's the year of publish, and you might have a text field that's the synopsis. So if that's a standard record for a book in if that's a standard document for a book in MongoDB and that's put in a collection, suppose now I want to create a vector index off of the synopsis, I could take that text synopsis text, I could pass it through an embedding model of my choice and I then store the array of floats that come back from that embedding model. And if I wanted to, suppose I had 10,000 record I had 10,000 documents in my collection, maybe I only wanna do a vector search on the top thousand. Maybe I don't wanna do it on all of them. Because of the flexibility flexibility of of the the schemas, schemas, because you can have different shapes, MongoDB not only allows that but thrives on it really well. So we had lexical search, we had vector search, and then that enabled us to do this notion of hybrid search. So suppose I wanted to do combine a lexical search of the title of the book with this now vector search that I have on the synopsis. And what if I also wanted to pre filter because I've got some integer fields in here. Suppose I wanna do a search there where I do a lexical search on the subject and a and a vector search on the synopsis, but I want to eliminate any documents, any books that weren't published after the that were public oh, I only wanna see books in my dataset that were published after the year 2000. So now I have these three levers of query power that together are give me lots of interesting ways now query my data that I combine this sort of pre filtering based on the metadata with the lexical search with the with the vector search. So once we had that in play by, like, 2023, we thought, well, this works with a embedding model you want, but could we make this easier to develop if we had an embedding model that was part of it? And that's why we purchased Voyage in 2025, which should create this better together story. We can get into all kinds of things with the better together, but the way that we go to market with Voyage is you can use Voyage on whatever vector database you want. You can use any embedding model and use MongoDB as your vector database or you can use them together and there's a better together story that's part of that. So that's kind of our history. Like I said, kind of unusual that it started with lexical search, but it kind of went lexical search, vector search, hybrid search with pre filtering, and now this better together storage with the Voyager Betting models. That's how that built over time for us. What's it like to query that today? Are you able to I I think this is like a has been relatively recently advanced. Right?

[29:10] Nathan Labenz: That I can now issue a single query and put kind of in SQL terms, I guess, a where clause and, like, all three of those different kinds of data and the engine under the hood, which filter to apply first, like, all that stuff kind of I don't have to worry about anymore. Is that right?

[29:28] Pete Johnson: That is.

[29:29] Nathan Labenz: So one of

[29:30] Pete Johnson: the advantages that we have in the way that we implemented vector search is it gets to stand on the shoulders of the base product that we already had. So we've been running Atlas in production for a decade now. So we already knew know how to replicate data across multiple data centers, even across multiple clouds. We already know how to do sharding, whether you wanna keep data close to this specific users for business reasons or for legal reasons. We already know how to do security, not only query at rest in or not only encrypted at rest and encrypted in transit, but we can also do during the query. We get to stand on the shoulders of all of that for the vector search. And one of the things that we get to do on top of that is MongoDB has always had this notion of aggregation pipelines. And the point of aggregation pipelines is instead of forcing you to do a query, get some JSON wad back of your result set and then have to massage that before you use it. Instead, with the aggregation pipelines, you can give extra you can give extra instructions to the back end so that that data gets sorted, filtered, massaged in different ways so that what you get back from that API call, you can now use immediately and you don't have to do any other further massaging. So because we already had that mechanic in the core product, what you're referring to is we just announced a pair of features. One's called break fusion and one's called score fusion. And what that allows you to do is in absence of an aggregation pipeline in these new aggregation pipeline stages, you would have to make the vector search in and then independently, you would have to make the lexical search and then you would have to combine them yourself. Now with those two aggregation pipeline stages, you can do it in one API call. So you only have one round trip call to our servers and then on the back end, we'll do both of those calls for you, and then we'll combine the results either based on their rank with rank fusion or based on their score with score fusion. So that what you get back is an already sorted list based on which one of those functions in the aggregation pipeline you chose so that you could then use that JSON WAD, that result set immediately. I'd love to learn a little bit more about how that works because this is something that I have run into,

[31:51] Nathan Labenz: and I've have have come up with pretty simple solutions for myself. Right? But, like, you sort of have the filter, and then I'll just be like, okay. I guess I'll sort by the vector match score subject to the filter. Right? Or we'll kind of tier them based on different filter values. It's always struck me that there should be smarter ways to do that that I've come up with, and I mean, you guys have some. So what are the kind of galaxy brained approaches for getting the best final sort order from these kind of fundamentally different things?

[32:28] Pete Johnson: So for now, what most people are doing is and this is the reason why we have the pipeline stages that we do. It's it's usually based on a rank or by score, and if you also throw in the pre filtering that lets you that lets you reduce the dataset before you do the queries, and then you get those queries back and you can them based on either score or rank. For now, that's what most people are trying to do. But my joke for this, Nathan, is there's two kinds of developers. There's those that love MongoDB and there's ones that haven't tried us. We're always in a pretty tight loop of feedback where we the whole reason we came up with Rank Fusion and ScoreFusion is because the community told us they wanted it. In very much the same way that why did we start doing Atlas search? Why didn't we make Lexical Search part of the platforms? Because we saw people were using it. We found a way we saw a way to make their lives easier. If it's so that you don't have to spend the operational cycles managing your own Lucene servers, what could you do with the cycles that we give back to you? And it's the same thing with rank fusion and score fusion so that you don't have to manage that code base yourself so that if you could just throw it in as a verb in the query that you already have okay. Now what are you gonna do with that with that time that we're giving back to you? So where it goes, community's gonna tell us what other kinds of things they want with that.

[33:50] Nathan Labenz: Read more novels is one candidate answer for what I'll do with my time savings. And going back to your book example, another thing I've struggled with at times is let's say I have the book, but let's say I actually have all the contents of the book too. Right now, I've got hundreds of pages of text associated with this kind of one entity. Now I can break that down into I just read a novel that had acts, so it had, like, four different acts, then, of course, there were chapters within the acts, and then there were sections perhaps and paragraphs, and, obviously, you're down to sentences at some point. How do you think about and I think there's some new work on this too, although I I confess I don't grok it fully. How do you think about, like, how that raw text should be chunked or otherwise divided up to be represented in vector space. Is it like am I looking for a single answer to that, or is it do you think of it more like a trunk and branch and leaf kind of thing? I've experimented with that kind of stuff a little bit as well, but I don't I guess I I I don't really know what the state of the art is in that realm today. Fair. So there's a couple of

[34:00]Deepgram Flux TTS: Deepgram Flux TTS brings lifelike AI voices with real personalities that handle interruptions, pauses, and natural conversation. Try all the voices free through September 12 at https://deepgram.com/keep-talking

[34:30]Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Main Episode

[37:20] Pete Johnson: more broadly speaking, and I promise I'll answer your question.

[37:23] Nathan Labenz: More broadly

[37:24] Pete Johnson: speaking, the use case for this level of search, for this kind of semantic or vector searching, is typically rag pipelines for agents or for a more sophisticated agentic memory for agents. That's when you typically see vector search start to enter the conversation when it comes into an agentic architecture in some way. Specifically, what you're talking about here is chunking and chunk size tends to be more in that rag pipeline use case. I've got some external data. I've got some proprietary data. I wanna inject that proprietary data into my LLM of choice without having to go through an expensive fine tuning or training process because the LLMs are trained on public data. They're not trained on my proprietary data. So how do I inject that in a safe way that still gives me some data fidelity so that I'm not I'm not giving away my proprietary data, but I can still get it to focus on my business problems instead of every LLM knows who the last 30 Oscar winners were, and that's not super helpful when I'm trying to get an agent to help me figure out what my knock should do next or my my IT ticketing should do next. So that's the use case. So if you've got a bunch of documentation that someone who is a human professional in that world, you now need to figure out how big of the how big that should the chunks be that I place in the individual documents and what's the fidelity of that. And when you're the application developer trying to figure this out, there's this trade off that you're looking at. So if I make my chunks too small, let's say to the fidelity of a sentence, if I make my I potentially lose the context in which that sentence appears. Just if you just read one sentence from a novel, doesn't tell you a whole lot. But if you then also read the paragraph or the page, it tells you a lot more about what the context of that sentence is. So that's an argument for, well, maybe I should make my chunks bigger. There comes a point though where if my chunks are too big, my storage costs start to go up and my retrieval quality starts to go down because if I've got three pages, let's say, and I'm trying to get to the fidelity of a sentence, I now have too much information and I'd lose the fidelity of that sentence. So what's the right answer? I I've been doing this a long time, Nathan. The answer is always it depends. Right? So you have to go through as a developer, you have to go through this iterative cycle like I chunk to a certain size. I run some tests. What's my retrieval quality? Okay. Now I try it again.

[40:10] Pete Johnson: I try a different chunk size. I do some tests. I see what my retrieval quality is, and I have to do that three or four times that for my use case to try to find out what the right balance is between chunk size, storage costs, and retrieval quality. So one of the reasons we bought Voyage is because, like, those guys like, their side hustle is teaching people at Stanford how to how to build LLMs. Those guys are all geniuses, and they came up with this new concept last summer called contextualized chunking. And we've got a blog article on this that we can potentially leak through here. But the idea here is that if left to your own devices and figuring out your own iterations of chunk sizes, there comes a point in a graph where if you've got retrieval quality on your x on your y axis and you've got chunk size on your x axis. With a traditional embedding model, there comes a point where at lower chunk sizes, you have zero context. So the retrieval quality is low, and it it builds. But then at some point, it flattens, and then it starts to degrade if your chunk sizes become too big, and that's why you have to go through these iterations to find the right combination. But what if you didn't? What if instead of sending that all as one big text blob, what if instead you send it as two? What if you sent the fidelity of the sentence you actually want and then as a second string, you send the size you send the other contextual informations where the name contextualized chunking comes from. And we will figure out for you what the right chunk size is for that combination and we will give you an vector array that comes back that balances those things for you. When you flip that script, it turns out you can get better retrieval quality with a smaller chunk size, which otherwise is not possible. If you're doing it the traditional way, you the way to get better chunk side way to get better retrieval quality was with higher chunk size. But if you use this contextualized chunking, we released version three of that last summer. We just released four of that in the last six weeks, you can flip that script. Now there still are use cases where you need to control the chunk size for different things. But if that's something that you wanted to not worry about or have to learn about, and the way I think about this is more developers will build AI agents in the next three years than did in the last three years. And in order to make that possible, we have to lower the learning curve. We have to make it easier for the developer ecosystem to learn how to do this. So if you don't wanna have to learn how to go through the iterations of figuring out what the right balance of chunk size and retrieval quality is instead, if you could use contextualize chunky, we'll figure it out for you and you still get good retrieval quality out of it. So that's that's one of the benefits that Voyage offers that no other embedding model out on the market offers, that ease of use feature that gets you better retrieval quality. I definitely appreciate not having to worry about it, but I do wanna learn a little more about it.

[42:56] Nathan Labenz: It it sounds like this happens in sort of a test free way at the level of the developer. So, like, I don't have to bring a bunch of eval suite work

[43:09] Pete Johnson: to

[43:09] Nathan Labenz: the table. So how the level of kind of principles, like, how is it working in the background? And you said also I'm getting one vector back. Right? So I pass in the sort of the chunk that I think I would really wanna be able to zero in on and context, and that gets converted into a single vector back that represents both of those things in, I guess, some sort of superposition?

[43:37] Pete Johnson: Yes. That's exactly how it works. So you get back one array of floats exactly how you did before. It's just now you don't have to go through the iterations of figuring out the size of the chunks you need. And similarly, there's another voyage feature that all the voyage all the voyage embedding models have called Retroska Riesling. Because the other place you have to make a decision about storage cost versus retrieval quality is with number of dimensions. So what do I mean by that? Everybody knows what two dimensions is if you've taken high school level algebra. Right? X y. Right? But in an embedding space, you tend to have at least two fifty six dimensions and sometimes as high as twenty forty eight. And the more dimensions you have, so each dimension is represented by one of those floats in that array of floats, the more dimensions you have, the richer your embedding space is and the better retrieval quality you get. But at higher dimensions, that doesn't come for free. You've got a storing 256 floats, takes up less space on disk and in the index and memory than it would for twenty forty eight. So you again have to go through this iteration of what's the right number of dimensions for my use case given my what my storage costs might be. All the voyage models have a feature in them called Betrovska reasoning. It comes from the Russian nesting dolls. If you think about how Russian nesting dolls work, right, you've got you've got one doll and you open it up and there's another one exactly the same, but smaller, a smaller fidelity inside, and then you keep doing that in over and over. So what what the how the how the voyage models work is when we generate let's suppose you did it ten twenty four. Suppose you wanted ten twenty four dimensions. You run some tests. Now you wanna try five twelve. With a traditional model, you have to run your entire corpus of data through a second time at five twelve. But with voyage models, you don't have to do that. When you've run it through at ten twenty four, vectors, those floats, they're ordered. So if you wanna try five twelve, you just lop off the last five twelve and that's now you immediately can begin testing on the remaining five twelve. So again, it doesn't completely solve the problem of figuring out what the right combination of retrieval quality and storage space is, but it helps you get to the answer faster. So these are the there when you take some of these things in sum, there we've got three or four of these kind of features that make it easier to use and help you get to your final answer more quickly. And the idea there is to give you time back in your day that you can work on your business logic instead of figuring out the plumbing.

[46:22] Nathan Labenz: Yeah. That's cool. I'm a huge fan of Matryzka anything. When does this stuff become necessary? So for me, I'm basically a business of one, and I try to be an early adopter of everything that I can. And I do have a pretty well working, I call it deep context, but basically a retrieval system that allows my agent to go into kind of all of my history from the last five years essentially, which is emails and Slack messages and everything I publish online and DMs across all kinds of channels. The podcast of the the transcript of the podcast, I should say, diarized so it knows what I've said and what the guest has said. It adds up to about a gigabyte in my case. Sure. I haven't really optimized it much at all. I just kinda let the agent throw it into a database of its

[47:18] Pete Johnson: choosing

[47:19] Nathan Labenz: and put whatever optimizations on it it felt like it needed to. And then we did get a at one point, was like, well, yeah, we could probably do better than keywords, so we've got an embedding layer on there as well. I think full disclosure, I believe I used the Gemini embedding model for that, but it's not very well optimized. How would I know if I'm really missing out on something? I I don't have, like, a huge eval suite. Right? It's just, I'm kinda vibing it with it doesn't seem to be working well. Is it a matter of, like, data scale, scale of users? Is it about, like, cost? I wanna optimize my inference cost, that's where I really need to get serious about how much data is is being returned. Like, what are the what are the thresholds that people or, obviously, larger organizations cross where they're like, okay. We can't really do it the let the agent choose its own adventure way anymore. We really need to try to get serious about some of these optimizations. Well, I'd return to the three things I mentioned

[48:20] Pete Johnson: in our first twenty minutes or so. It's it's speed, scale, and retrieval quality are the main three things. What most people do, and you're not gonna offend me if this is what you did, I mean, most people start with Postgres and pg vector and then they choose they choose their embedding model with whatever cloud they're using. Like Gemini is prominent if you're gonna if you're gonna be on Google just like OpenAI's embedding model is is pretty popular over on Azure because of the historic relationship that those two companies have. But there comes a point in when you're when you're doing a demo, when you're doing a POC, it doesn't always show itself. But there comes a point where when do milliseconds matter to your use case? When does scale matter to your use case? And that typically comes depending upon chunk size, that typically comes at about a 100,000 vectors is when is what I mean by scale. And when does retrieval quality matter? If you look at Voyage AI model, Hugging Face has a benchmark out there called RTEB that Voyage AI models are typically at the top of, and we can get as much as 14% improvement compared to some of those other embedding models that we just mentioned. So are there use cases for which a 14% difference in embedding model quality is the difference between a hallucination and a correct answer? And that's before you even start putting re rankers on it, is another way that you can boost retrieval quality without having to to do anything special to your data. So like I said, it's it's speed, it's scale, and by scale, I typically mean in the neighborhood of a 100,000 vectors and retrieval quality out of your embedding model. Most people think embedding models are commoditized and that is not true. There is a very big difference that you can get in retrieval quality based on what embedding model you choose. Anthropic does not have an embedding model in market. They recommend us.

[50:15] Nathan Labenz: It's a great recommendation. You mentioned re rankers, and and then there also this kind of earlier concept of sorting results out of the database. This brings to mind this concept of bitter lesson engineering that I think is kind of growing in prominence, which from my simple point of view is just like every so often, you should probably kinda go through your stack and look at all the kludgy extra things that you did to make things work and say, like, which of these do I no longer need because the model got smarter or the embedding model got better? And so things are just kind of naturally working now or could naturally work now where in the past I had to kinda do all these artisanal craft sort of things to to make sure that they worked. Are you seeing examples of that in the retrieval space broadly where things are in some ways getting easier, or have you not really seen the bitter lesson applied to these pipelines, these production environments?

[51:23] Pete Johnson: Yeah. I think there's two places that I've seen that recently, and one of them is with the re ranking that we were just talking about. So in this free, we released dollar re rank, which is a companion to the ScoreFusion and rank fusion that we already talked about. So those two relate to doing a hybrid search. The dollar re ranked, again, you would in a historical way, if you would have to do your vector search once, send your results to a re ranker to get them reordered in a way that is most optimal for a rag use case to then pop them into the context window. Just like we did with the score fusion and the rank fusion. We've now got a stage where if you do dollar re rank, you call the API once. We'll do both of them for you on the back end so that you only have to make one round trip to the server for that. So that's one place that we've seen some additional ease of use. The other one is a feature that we released recently called auto embeddings, which is you tell us this is a better together story. You tell us what which collection, you tell us which attribute on documents in that collection, and you tell us which voyage model and how many dimensions you want, and we'll take care of the rest of it. Anytime an existing document with that attribute in it changes, we will automatically take that new let's say it's text, put it through the embedding model, update the vector in the document, update the index in memory, and we'll do all that for you. If a new document shows up into that collection that has that attribute, we'll go through the same cycle for you. So, again, trying to remove some of the plumbing so that you get some time cycles back as a developer so you don't have to you don't have to go craft your own and maintain over time your own embedding pipeline. You just pass us one JSON wad, and we'll do it for you.

[53:18] Nathan Labenz: Zooming out a little bit from the database to the the the broader systems that wrap around them, you stuff about memory for AI systems. And this is another thing where I've seen kind of the great cation and then somewhat simplification. Probably simplification has been enabled by things that enterprises are maybe a little reluctant to pay for at scale. Like, when I get a million token context window, I'm like, great. Now I don't have to worry about all this stuff. I can let the model handle a lot more of it. But I do take your point that, obviously, you got thousands of users that can add up. How what would you describe as and, of course, it depends, but trying to abstract away from some of the details and give me kind of a center of the distribution description, what would you say is, like, the state of the art in what a good memory system looks like today? Sure.

[54:16] Pete Johnson: So I again, history matters here. Right? When ChatGPT came out in 2022, in the end of twenty twenty two, it had a very simple application architecture. You took you took the query, you put it in a context window, the LLM would process it, and give you a result. End of story. That's how it worked. And by spring of that year by the next spring, you would see headlines like ChatGPT passes the bar exam, and it would have these very cool things it could do. And we've already talked about why you need RAG because those LLMs even today are only trained on public data. If you wanna inject proprietary data into it, you need to to specifically do that with a RAG pipeline. What we saw in 2025, though, was the second problem that we see with with the core LLMs, and that gave rise to the need of memory. And that is, like, in spring of twenty three, if you had asked ChatGPT what's MongoDB stock price today, it couldn't tell you. And the reason it couldn't tell you was the LLMs have a knowledge cutoff date. There's a date after which they don't know anything about the world because the model vendors at some point have to stop training and put them into inferencing mode. And the way we overcame that in 2025 or so was with tools and with MCP and being able to have to grant to the LLM, hey. Here's this API or here's this tool we could call to go get additional information. That's why whether you're a Claude person or a Gemini person or a ChatGPT person, that's why today, if you ask, hey. What's MongoDB stock price? It can tell you because it can find a tool in its arsenal. Go make a web search, get that information, and report it back to you. The other thing we saw in 2025 though is this notion of looping where the result of one loop of that agent then became the input of the last one. The context window is still very dumb. Right? The context window is still you have a fresh context window with every time you pass it to the LLM, and that's when we needed agentic memory. And the early passes we saw of agentic memory was short term memory was typically, okay. Take every response that we got within this session and cram it into the context window so that we can see what happened earlier in this conversation. And then long term memory was, okay. Now do the same thing for every session we've had for the last three days and cram that in there as well. And then a two negative side effects that I now see people starting to push against. Negative side effect one, we already talked about is token maxing. Right? If you if every agentic loop puts a million tokens in the context window, even con even context you don't may may necessarily need, your cost starts to add up. But there's also a functional cost to that. There's a couple academic studies that show that, like, the first seven k and, like, the last seven k tokens are the most important, and the stuff that's in between can end up just muddying and confusing the LLM as it's trying to come up with an answer. So instead of thinking of how could I maximize, how could I cram a 100 a million tokens into the context window every time, what if I could choose just the right 200,000 for this agentic loop? So we're starting to see more sophisticated short term memory that doesn't take everything in the session, but just the things in the session that are relevant for this pass. And the same thing with long term, and we're starting to see some new memory types, especially in fortune five hundreds. There's a new memory type called taxonomic memory. So think about either in a vertical, so like automotive we were talking about

[57:47] Nathan Labenz: before,

[57:48] Pete Johnson: or in a specific company. There's some terminology that is specific to verticals or specific to a particular company that a layperson who is not in that vertical, those words may mean different things. So you might have a list of, let's say, a 100 of those terms. But for this agentic loop, what are the five that are relevant for this loop of the agent? And then do that again for the next loop. What are the five that are relevant for that loop of the agent? So again, instead of taking all 100 terms and putting them in the context window every time when not all of them are gonna be relevant, what if you could instead have a more sophisticated memory type memory system that instead of just taking everything, used vector search or semantic search or some combination of vector and lexical with some pre filtering and just got the just the right just there's some perfect you never get there, but there's some perfect context window for this pass, and that perfect window doesn't use all of it. It uses just the stuff that it needs to. Those are the kinds of things that we're starting to see in enterprises both to combat the cost of the tokens and to combat the relevance of the tokens so you can get the best answer out of this call to the LLM. And do have any tips for maintaining

[59:12] Nathan Labenz: these memory systems? I mean, this kinda goes back to some of the challenges with, and may maybe there's good solves within MongoDB for some of this. If so, definitely highlight them for me. But I sort of struggle with the lack of normalization or my failure to observe the dry rule in my own memory system sometimes. I'll have, like you know, I have these, like, raw exports that combine my email and DMs and everything else right into these kind of

[59:44] Pete Johnson: Sure.

[59:44] Nathan Labenz: Monthly log files. And these are typically, like, a couple 100,000 tokens for a month. Then I'll, like, summarize those into a monthly summary, And then I'll take the monthly summaries, and I'll put those into a yearly summary. And then I've got, like, a wiki type Carpathi inspired structure that sits on top of all that and is, more entity based, they kinda link to each other. And some of this lives in a database, some of it lives in a file system. But now I've got a challenge where I'm like, some new stuff has happened.

[1:00:15] Pete Johnson: Okay.

[1:00:16] Nathan Labenz: What do I do? What do I change? How do I make sure that it's changed everywhere it needs to be changed? I sometimes also have instances where I've had really funny I mean, I think the models are getting better with this, but it's less of a problem these days. But in setting up in the first place, I had some problems where the models would see a project that I, like, experimented with a little bit or, like, was maybe gonna pursue and then didn't. And it will, like, keep it as, like, an open thread for months and months. And I'm like, dude, never actually did that. You just forget about it. Forgetting is also sometimes real virtue, memory system. How what have you learned? What tips would you offer in terms of how to maintain these things so they're accurate, not just when you first create it, but on an ongoing basis?

[1:01:05] Pete Johnson: This is the hardest problem with agents right now, I think. And the way that I put this to people is we as an industry, we've been building databases for sixty years.

[1:01:15] Nathan Labenz: We've been building agents

[1:01:16] Pete Johnson: for, like, eighteen months, man. Like, nobody knows all the answers yet. We're all sort of collectively figuring it out. But what I do see with these more sophisticated memory types is that as the application author, that you have two responsibilities with these memory systems, where with the prior one, you only had one really. So not only do you call this memory system, hey. Here's my query. Give me the best maybe you give it a token budget is the way some of these work so that you don't you don't have to figure out how many memory types there are and how to query them. You're just like, give me the best 50 or give me the best 100. You give it a token budget, and then it gets it back from the mix of different memory types it might have. And then you put that in the context window, you get your answer, and then your second responsibility is to send that answer back to the memory system. So the memory system can then curate it and then write it into the memory system. And when it writes it into the memory system, the more sophisticated ones also have this notion of RBAC. So that if you and I are in the same job type that we get to share the memories, so that if you come up with a really good memory and then I get to reuse it later, then that ends up being beneficial for both of us. So that's the second responsibility you have. But the hardest part is what you're talking about here. The way that one coworker just wrote this to me yesterday, I wanna quote him. Write, change, recall, forget. Because these things have a a half life to them. Right? Like, things that are more recent are more important than ones that that took place weeks, if not months ago.

[1:02:50] Nathan Labenz: Maybe you can tell me about any, like, Mongo specific tricks or advantages for these sort of graph structures that I seem to keep making. I have people and the organizations they work for and the ideas that that I sort of associate them with, and these things all kind of point to each other, but they do it in a pretty loose way where I'm I'm right now kind of trusting the agent to hopefully notice those pointers and to the degree I'm, like, running maintenance. Hopefully, follow those pointers and do the necessary maintenance. I don't have a lot of guarantees, and I suspect that there are better technologies that I could be building on that would give me a lot more robustness. Well, there's there's a couple different ways to tackle this. Like I said, the

[1:03:40] Pete Johnson: forget part is the hard part of it. But what we see our customers doing is this is where, like, the retrieval quality of those memory types, that's where that that non commoditized embedding model can make a difference and where a re ranker can make a difference. You you can get depending upon the use case, you can get like a five to 10% boost at your retrieval quality just by using a re ranker on top of whatever your embedding model of choice is. And that's true of architecturally, that's why we made it so easy to implement the re ranker on top of the voyage embedding models when you're doing that vector search. But there are use cases where maybe you have so much data that you might have to take a hybrid approach where maybe you use maybe use a graph structure for, I don't know, two to six levels of data and then you get down to a leaf and you do a vector retrieval inside that leaf. We see people doing that as well. There's all kinds of information you can find on our website about how you can because we're JSON based, you can use MongoDB to build graph structures into your data so that you only have to go one place for a graph database, a core database, a vector database, the embedding, re ranking all in one place so you don't have to try to stitch together multiple tools yourself and have to maintain that over time. But we do see people doing that with graph structures. It's typically not as deep as, like, you you would traditionally think of as a as a graph database need. But like I said, it it's pretty common to go, I don't know, half a dozen or maybe maybe a dozen layers that maybe for bigger corpuses of data that are more heavily categorized, like some of our retail customers do that with product databases. If you think about how a hierarchy of products might appear on a website or on a mobile app, they might segment that first by product category using a using a more of a graph style. And then once you get to a particular product category, then do some vector searching within that individual category node. Are there any applications that you would point to as just being great examples of memory well implemented?

[1:05:51] Nathan Labenz: Well, our biggest customer right now, they haven't been very public about how they did it. There's a company called Eleven Labs out there. They have multiple agents per customer. This sort of micro agent approach where you get multiple smaller agents at the disposal for individual customers and given the number of customers they have. So they Eleven Lives started life as a model provider that was doing way sophisticated speech to text and text to speech, and they built a platform on top of that that is more of like an audio editing suite that they do for that. And as part of that, they they have a bunch of agents doing all kinds of editing and transformation kinds of things for their customers. And if you think about the kind of context memory that they have there, an important way to to do that as well, I would worry about any memory architecture that instead of relying on lower costs, embedders, and re rankers is relying on multiple passes of the LLM to help you categorize and shrink the the the corpus of data that you might wanna then put into the context window for the larger sort of more functional LLM call because then that's just adding up tokens as well. I mean, that why things like embedders and re rankers have a lower token cost is, again, right tool for the right problem. Yeah. That adds a lot of latency too at this point. Even the small models can reason for quite some time before you actually get your answer from them. They can. And so all those things add up over time. And it's it's not just tokens, but you're you're right. The overall latency for the decision loop that you're into makes a big difference. I think you also have some interesting takes on build versus buy analysis. I won't even try to summarize it. Just give me your your hot takes on how people should be thinking about building versus buying. What is mature enough in the AI realm to buy and what do you even if you kind of buy a part of it, what do you still have to expect that you're gonna end up building or customizing enough that it kind of feels like building? Yeah. So I have the good fortune in my job. I've been to seven countries this year to talk to probably a 100 different customers about where they are in their AI journey, and they tend to fall fall into sort of three camps. Camp one is I bought a license for this one tool and I'm done. Right? Like, my AI strategy is done. I bought one thing. And while that could be a good starting point, typically, is not specific enough to help solve the problems of your overall enterprise. Then you have a group of people who have done some POCs and maybe a couple of production deployments with varying degrees of success on ROI that had struggled with ROI. I always argue probably picked the wrong problem. Picking the right problem is really important in this space. The way that I encourage customers to think about this is, like, what are the top 10 to 15 problems that you have going on in your business right now? Of those, what do you have good data for? You made a comment earlier in this conversation, like, quality is a big deal. Things like bad data quality and bad security posture don't get solved by AI. They get amplified by AI. So what's your what's your biggest problems? What do you have good data for? And then what do you already have metrics for around the problem? And that's the biggest difference that people tend to skip over when it comes to problem selection. Because if you don't already have metrics for how something is performing, you won't know if it got better. So call center use cases are very popular low hanging fruit in enterprises because I already know what cost per call is. I already know what call volume is based on how I'm already bonusing people in those jobs. So if I introduce AI into their workflow and I see those numbers jump, I can attribute that change to the AI and I can do a back of the envelope ROI. We see the same thing with software delivery life cycle. Early in the year, we saw all kinds of bragging about, well, now because I've got Claude code or I've got t gravity or if I got codex, now I can produce five times as much code as I did before. Anybody who's been doing it any length of time knows that lines of code is a terrible metric

[1:10:04] Pete Johnson: to judge the productivity of a set of developers off of. Instead, how how fast are you getting from idea to production deployment? That's the metric that matters there, not not necessarily lines of code. So the metrics there matter, and that's the difference between the folks that are in that middle state either stuck in sort of POC purgatory and haven't quite got to production deployments. The biggest difference isn't in how they're applying the tech. It's what problems they chose to try to solve. And then you've got people that are sort of more advanced that are looking at some of these more sophisticated memory types and things where they're trying to optimize the systems that they have. And in some cases, they're they're making purchases of larger platforms to help them do that. Other times, they're doing homegrown. And like I said, we're we're like eighteen, twenty four months into this. There's there's no there's no one way to do this here. There's there's no lamp stack for agents yet in the way that we have with web development. We will get there. Having lived through that having lived through that life cycle, we'll eventually get there. But, like, there's there's no React in Angular. There's no LAMP stack for agents right now.

[1:11:12] Nathan Labenz: You mentioned call centers, and this is the second time today that this has come to mind. I just ordered a pizza last Friday night from the local pizza place in my neighborhood. Shout out once again to Greg's Pizza in Detroit. I think it's just a one location business. And who answers the phone? Style pizza is underrated, by the way. It is. This actually is a a traditional circular round pizza, but I agree. After auto, it's Detroit's biggest export to the world is pizza. But even just this one location business, right, I call and who answers but an AI agent. And it was a pretty impressive experience, I have to say. Like, very natural conversation, not flawless. There were a couple little things where especially as I started to get a little adversarial in my testing of it. Like, I asked it what company is powering this AI agent experience that I'm having, and it took that as it somehow thought I was ordering pizza for my company. So, you know, there's some kinks still. But I overall was, like, very impressed. Like, the happy path I thought worked really well. And this has me thinking, boy, there must be just a ton of transformation happening at call center businesses. How how much are you seeing, like there's always this kind of waves of change, right, where, like, the first wave is, we give our agents a tool where they can, like, get help faster and be more accurate. But then there's this, like, next wave or, you know, maybe several waves, but certainly one that seems like it's coming is you just call and talk to AIs. Right? How far along that journey are the customers that you are talking to? How much, like, actual disruption have you seen in that part of people's businesses so far?

[1:12:57] Pete Johnson: Most of Fortune five hundreds that I talked to are Dewey employee facing use cases with human in the loop for exactly the reasons that you just laid out and for some of the ROI reasons that I laid out before.

[1:13:10] Nathan Labenz: So

[1:13:10] Pete Johnson: if you think about the employee facing use case, you already know what you're bonusing your different employees are and what KPIs, what key performance indicators you're using to judge whether or not they're doing a good job. So like I said, I used the example of call centers before, but you can think of any job whether you're an insurance adjuster or a line worker in a manufacturing plant or whatever your job might be. Everybody's got some key performance indicators. And if you introduce AI into whatever that workflow is and you see those jump, that means that you're you're you figured out some ROI. But there's also security a data security issue for that as well. If you and I are employees for the same company, is it great if my salary accidentally leaks to you? No. But it's way better to have that than if we are two customers of the same company and I get to see your data. That is a VP has to get on blame for that kind of scenario. So, like, the risk reward is higher for customer facing use cases than it is for employee facing use cases. So at least where we are now in the ecosystem where I mostly see the Fortune 500, you with exceptions, of course, but I see a lot more work going into those employee facing use cases with human in the loop than I see going full autonomous AI with customer facing use cases. For you and and what you're doing at and for MongoDB, are you you've of course, you've got traditional documentation, right, that human developers can But go and I'm sure you're also thinking a lot about how do you make your documentation AI friendly, and then how do you have an AI that can proxy into that documentation? And then potentially, you're, like, packaging up and distributing skills and trying to think about, like, how to become the database that is, like, recommended by the agents when somebody's doing something for the first time. How is your role as bringing this technology to the world evolved in light of all the different ways that people are discovering and starting their implementation process? So all the things you just mentioned are certainly things that not just us, but every software company is involved in one way or another. The two that I would shine spotlight on are the MCP tools to make it easier for agents to talk to your data in your MongoDB instances. We had an MCP server for a while and we were now kind of self hosting some of those as part of Atlas as part part of the ecosystem. And then the other one is the agent skills. So think of that as kind of curated system prompts for doing things like data modeling, doing things like optimizing operations, being things that we have well organized playbooks for that previously we would have as documentation, but now we have them as a series of doc of markdown files that you can feed to an agent of your choice to give it sort of the lowdown on different ways that it should approach different tasks. And there's maybe the I think it's agent skills we just dropped a couple of months ago has like the top eight. It's like it's six to eight different agent skills that we have for things like operations and data modeling to help you do those sorts of things. So, yeah, that's become pretty commonplace these days, and we're certainly part of that ecosystem.

[1:16:32] Nathan Labenz: You mentioned this acquisition that MongoDB did of Voyage a couple of times, and I had a kind of just a couple, like, theoretical questions about that.

[1:16:45] Pete Johnson: Okay.

[1:16:45] Nathan Labenz: One is it's striking that Mongo built its own VectorDB. Right? And there's, like, a bunch of VectorDB startups out there. So, like, one could have imagined an alternative scenario where there was an acquisition of a VectorDB startup, but there wasn't. There was a acquisition of the model startup. Then also just look at the relative value, and you said these guys are geniuses, and I I don't doubt that. But the price at even at $220,000,000, which is I'm old enough to remember when that was serious money. You know, it's it's less than a tenth of a percent of MongoDB's overall market cap. So I'm like or less than 1%, I should say. So I'm just kind of wondering, like, what should we infer from these observations about value and defensibility in the software business over whatever passes for the long term in your mind? You know, you could tell a story about, like, infrastructure is the big winner, models get commoditized, incumbents can defend themselves against startups. Like, what do you think are the right macro lessons to draw from this experience?

[1:18:01] Pete Johnson: I mean, fundamentally, what we've always been about has been how can we make the day in a life of a developer easier so we can make it easier for them to build their business logic and spend less time worrying about the plumbing lower in the stack. I went through a description earlier in this conversation about how we implemented vector search. And for us, because we already had the JSON based, it was it was relatively straightforward for us to just, okay, you add an additional attribute. That attribute is your array of floats. You generate an index based on that array of floats. So that was pretty straightforward for us to add to the existing product and take advantage of the things like the charting, the security, and the data replication that we already had in the base product that some of the net new vector database companies out there have to catch up and struggle with a little bit. So that's why we chose to sort of do vector search the way that we did is because of the flexibility based into built into the document model from the very beginning, it was it was pretty straightforward for us to do that. But where it was not as straightforward and why we made the Voyage acquisition was today, can still use whatever embedding model you want as long as it generates that array of floats. You put that array of floats to your Document, you go create your index, and you're off and running. But we saw an opportunity to to a, that the market was seeing embeddings and re rankers as a commodity, whereas we we saw Voyage really standing out. And like I said before, Anthropic agrees with us. And we could create these better together stories like the auto embeddings that I mentioned before. Another one I didn't mention is being able in Atlas to manage all your API keys in one place so that you don't have to go to one console for your core data, another console for your vector data, a third console for your embedding models, and a fourth console for your re rankers. You can get all that in one place. The management of it is easier over time. So if you look at the history of MongoDB features and really what our focus has been, we're all developers at heart and it's it's about making it easier for the developer ecosystem to learn and operate this these new techniques that we see in application architectures around agents, whether it's a Rag pipeline or whether it's the agentic memory. So how how do you make how do you make vector search more approachable? How do you reduce that lower lower that learning curve so that more people can learn it more quickly and start to participate in this ecosystem? Another story that I've heard that I wanna just get your reaction to and see if think it's true in your experience is I was speaking to a founder of a VectorDB startup one time, and I kinda made the case or put put it to him that, jeez, it seems to me like the incumbents are going to be able to add a vector

[1:20:53] Nathan Labenz: aspect to what they're doing before you're gonna be able to kind of replace everything that they're doing. And so it seems like they're gonna have a hard you're gonna have a hard time, like, really displacing them. And his answer was, well, that may be true, but most of the data that is coming into our vector database has never been in a database before at all. Right? It's just been sitting out in some data lake or data warehouse or just kind of unstructured piles of documents. And now it's coming into a higher level of infrastructure and it's being made more valuable in a way that just wasn't having at all before. Sure. Are you seeing something like that? And and what does that look like? I also noticed that there's the multimodal embedding model that supports video. So video would be potentially a great candidate for the kind of thing that has never been in a database before. Are you seeing this, like, phase shift of just, like, a much greater universe of data coming into Mongo than in previous eras? We are see

[1:21:52] Pete Johnson: we are seeing this broader ecosystem of data that that wasn't indexed before because it didn't lend itself to a traditional lexical search. The the cool thing about the vector search is I mean, vector search at its core is taking some piece of data and mapping it into n space, into some geographical geometric space. And then really all vector search is similarity, like where what are the closest vectors to this new thing that I'm searching on? And video is a great example. Audio is a great example. Unstructured data, you just take a bunch of PDFs that you have sitting around in a SharePoint. I mean, those are good examples as well. So, yeah, I think that there's there's some truth to that that we're it it's not just how do I take the data and put it into a database, but now how do I find it? And how do I find in a way that is fast and scalable, that's a good quality? And that's why the way that we implemented vector search and being able to within the same platform have the combination of the pre filtering, the vector search, and the lexical search. You feel like that gives us advantage and gives our developer community more levers to pull from than what some of the alternatives are. And like I said before, be because it's it stands on the shoulders of the core product with the with the Atlas version, you can already deploy that in any hyperstellar data center you want with good data resiliency, with good security, and if you need to to shard that data so that it doesn't meet particular geographies. We got all that for for essentially for free because of how we implemented vector search on top of the core product. And, yeah, we're we're seeing all kinds of different kinds of data get put into those documents in a way that we didn't before.

[1:23:38] Nathan Labenz: Mentioned travels, think, so far to seven countries this year where I'm I'm so kind of myopically focused on what's going on in San Francisco and Silicon Valley that I'm mindful that I may be missing important stories or differences in perspective that are going on around the world. I try to fill that gap at least with when it comes to China, but it's a big world out there. What has stood out to you in your travels this year in terms of differences of perspective on AI usage patterns, values, you name it, could be anything, but just kind of what do you think The US audience in our inward looking way is missing that the rest of the world is doing? I think biggest thing if I turn question on its head

[1:24:24] Pete Johnson: just a little bit, there's a presumption in other countries that The US is ahead and doing things that other people are not, and I found the opposite to be true. So, I mean, I live and work in out of Cincinnati. So US is one of those countries. I've spent some time in both Amsterdam and London earlier in the year, but I just did a tour that included stops in Toronto, Bingaluru, Mexico City, and Sao Paulo. And the two most sophisticated customers I talked to this year were in Mexico City and Sao Paulo. And they assumed when we started their conversation that they had US competitors that were doing things that they weren't. And like I said, the opposite was true. I think we've reached a point with some of these technologies that, like, geographic barriers don't matter nearly as much as they did during, like, the web app era or or even during the cloud era. Because for the cloud era, like if one of the hyperscalers didn't have a data center in your country yet, you were kinda out of luck. That's not true anymore. Like pretty much every country has at least one hyper scaler data center in it and by extension access to models and vector databases and embeddings and re rankers. Like, access for these things is far better than what I've seen with previous technology revolutions that we've seen. So I think understand why there would be an assumption that US companies would be ahead since so many of the bigger AI companies are US based. But like I said, the two coolest things I've seen this year were in Mexico City and in Sao Paulo. So I think that those geographic barriers to being ahead in the market are starting to disappear. What do you think

[1:26:16] Nathan Labenz: are the barriers? Why is that, I guess? Is it that the American companies are culturally too conservative to run as fast as some of these international companies are? Is it that is it like a leapfrog story where they sort of had the international companies, I mean, had kind of less recent technology investment that they would have to get comfortable replacing? Or what's what's driving that surprising observation?

[1:26:45] Pete Johnson: Well, I think it has more to do with the lack of barriers outside The US than it does with any of the behaviors of the companies within The US. It's just access is democratized now in a way that it wasn't that whether it wasn't for the cloud and for the mobile waves. Right? Like, if you didn't have the radio towers yet, an iPhone's not gonna do you much good. But now we have all this infrastructure that's laid worldwide in a way that we didn't fifteen years ago that that makes access to these things far easier than they were before. So there's just there's just a wider set of people in a wider set of countries that have access now that they didn't before. So I I don't think it's any it's not an indictment of the way that I think US companies are thinking about it. I just think that there that greater access erodes those geographic barriers that we've seen with previous technology waves.

[1:27:35] Nathan Labenz: Any

[1:27:35] Pete Johnson: closing thoughts?

[1:27:37] Nathan Labenz: Any anything you'd wanna leave people with or anything I didn't get to that you would wanna make sure we touch on? Well, there's one technology thing that we have I didn't get a chance to talk about, and that's shared embedding spaces. And then there's there's also this notion of just, like, kinda where we are on this journey together. The shared embedding spaces real quick is one of the other ease of use things that the voyage models provide that nobody else does is in January, we released four versions of the of the version

[1:28:08] Pete Johnson: four of the text model. Everybody typically releases small, medium, and large, and it those different price the each one has a different price point and a different retrieval quality that comes wrong with that. But we also introduced something called the Nano, which is open weight and free for anybody to download in off of Huggiespace. And what shared embedding spaces does is each one of those four models shares an embedding space, meaning that embeddings that are generated with one of those models, they're compatible with the other three. And what they'd let you set up is you could take your corpus of data and you could embed it using say the large and developments. Every other embedding model is gonna force you to pay tokens during your development cycle to hit against their embedding model. So as your query as you're taking your queries and you're you're passing them through the embedding model to do your searches. But with us, you can use the Nano there. You can run that on your laptop if you wanted to. So you could if you wanted to have zero token costs aside from the embedding of the corpus of data. But for your queries, you could eliminate your your token costs during development in a way that wasn't possible before January. So that's not for everybody. You do take a little bit of a retrieval hit when you use a different retrieval quality hit when you use a different model. But it for some customers that don't wanna have to that wanna lower their token cost during development, this is one way that they can potentially do that. So that's that's shared embedding spaces which we introduced in January. And more broadly, I wanna come back to the idea that we've been building databases for a really long time. We have not been building agents for a very long time. We're we're gonna continue to see iterative improvement and it's gonna happen fast. Like, m c published m c p on, like, Monday before Thanksgiving two years ago, and by March, all of their competitors had embraced it as a protocol. Token Maxine first got mentioned in, like, early March, and by April, it was over as, a conversation piece. Like, the quickness, the speed at which we have these conversations and we go through these cycles of these different features, it just it's faster than it ever has been before. So the fun part is there's still a lot we all have collectively to learn. We're gonna all collectively figure out how to make this easier for everybody, and you you'll see more and more agents in everyday life as a result.

[1:30:37] Nathan Labenz: Well, as field CTO of AI at MongoDB, I know you're gonna be racing as hard as you can to keep up with all these cycles, and we'll continue to look for new updates and more great progress from you. Pete Johnson, thank you for being part of the Cognitive Revolution.

Outro

[1:34:41] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.

Published in: Episode

Read next

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to The Cognitive Revolution.

Your link has expired.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.