Intentional Tech: Designing AI for Human Flourishing | Alex Komoroske, Cofounder and CEO of Common Tools

Summary

Podcast Summary: "Intentional Tech: Designing AI for Human Flourishing"

AI and I with Dan Shipper | Guest: Alex Komoroske, Cofounder & CEO of Common Tools
Date: July 9, 2025

---

Episode Overview

This episode explores the future of AI technology with a focus on designing systems that promote human flourishing. Host Dan Shipper interviews Alex Komoroske, cofounder and CEO of Common Tools, delving into how Large Language Models (LLMs) can be used not just for engagement and profit but as intentional tools that amplify agency, privacy, and personal growth. The conversation critically examines the technical and societal architectures underpinning AI, challenges with current paradigms, and offers a systems-thinking perspective on building AI aligned with user intentions rather than corporate incentives.

---

Key Discussion Points & Insights

1. Reframing Tech’s Trajectory: Human Flourishing vs. Engagement Maximization

• Alex's Core Concern: LLMs are as transformative as the printing press, electricity, or the internet, but their trajectory depends on the intentions behind their design and deployment ([00:00], [06:46], [56:53]).
  • Quote (Alex, 00:00 & 06:46):
    “We have a choice. We can go down the path we've been going down—which is engagement-maximizing hyper aggregation ... Or we could enable a new era of human flourishing. That could lead to a new era of human flourishing… We want technology that aligns with our intentions—not necessarily what I want in the moment, but what I intend to do.”

2. From Chatbots to Coactive Systems

• Paradigm Problem: Chatbots are a starting point, but their 'append-only' conversational structure lacks the capacity for long-term, structured context ([00:54], [03:03], [04:24]).
  • Quote (Alex, 04:24):
    “Chatbots to me feel like a feature, not a paradigm... For long-lived tasks, you need structure.”
• Coactive Fabric: Alex describes Common Tools as aiming to build a “coactive fabric for your digital life”—a system where both user and AI agent develop meaning in parallel and context is shared, rather than siloed.
  • Quote (Alex, 03:03):
    “I think of it like a coactive fabric for your digital life. You are active in the system and so is this emergent intelligent process … your private intelligence, powered by LLMs.”

3. Intentional Tech: Aligning AI with Human Values

• AI & Personal Intentions: Emphasizes the opportunity for AI to learn and amplify stated intentions, not just revealed preferences (e.g., time with family, intellectual challenge), marking a departure from social media’s outrage/maximal engagement design ([06:46], [56:53]).
  • Quote (Alex, 06:46):
    “I intend to spend quality time with my family. I intend to experience new things ... Technology that aligns with your intentions—not just your dopamine-driven behaviors.”
• Personal vs Corporate Alignment: Warns that ‘personal’ AI from companies with engagement business models is inherently conflicted ([08:51], [16:31]).
  • Quote (Alex, 08:51):
    “The very first word—personal—doesn’t actually align … If you’re maintaining a dossier on me and that dossier leads to a powerful and proactive thing, that's terrifying.”

4. Architectural Foundations: Context, Privacy, and the Same-Origin Paradigm

• Stateless vs. Contextual AI: Most LLMs are stateless; true value comes from the user context layer ([09:29]).
• Security Models as Drivers of Centralization: Current “same-origin paradigm” (web/app security model) hinders interoperability and integration, covertly leading to feature centralization and monopolistic dynamics ([25:52], [29:19]).
  • Quote (Alex, 25:52 & 29:19):
    “The current laws of physics, the security model we use for the web and apps, actually limits this possibility ... Data accumulates inside that origin as a little island.”
• Open Attested Runtimes & Confidential Compute: Advocates for cloud-based architectures verified via confidential computing to ensure privacy without local-first inconveniences and to break the current triangle of trade-offs ([32:01]).
  • Quote (Alex, 32:01):
    “Confidential Compute is secure enclaves in the cloud … even someone with physical access to the machine can’t peek inside.”

5. The Dynamics of Open vs. Closed Systems

• Lessons from Internet History: Alex draws parallels between current LLM ecosystems and early internet/AOL vs. open web, warning against the dangers of repeating closed aggregative structures ([18:23], [19:27], [22:29]).
  • Quote (Alex, 19:27 & 20:15):
    “AOL was an important company… but eventually the open-endedness of the web took over … Open systems tend to win under certain conditions, especially in the growth era.”
• Feature Innovation Limits: Big companies ignore “50,000 user” features—this stifles innovation ([24:44]).
• Enabling Open Ecosystems: Open APIs and commodity LLM services are positive signs, but security and privacy must be actively architected, not assumed ([25:21]).

6. Systemic Thinking: Coordination, Power, and Evolution in Tech

• Complex Systems and Coordination Costs: The exponential increase in organizational coordination costs hinders innovation in large organizations ([44:44]).
  • Quote (Alex, 44:44):
    "As organizations get larger, they get much, much, much slower. And that's true even if you assume everybody is actively good at what they do. It arises due to an exponential coordination cost blow up."
• System Intervention Mindset: Lasting change comes from shifting leverage points (technical or otherwise) rather than brute regulatory force ([41:05]).
• Emergent Knowledge Transfer: LLMs unlock the fluid transfer of tacit knowledge, not just explicit explanations, enabling richer interpersonal exchanges (“liquid media”) ([49:04]).

7. AI Agents, Prompt Injection, and Trust

• Prompt Injection Risk: Compares the risks in LLM prompt injection to traditional code injection—except it’s even riskier due to “all text being executable” ([62:21]).
  • Quote (Alex, 62:21):
    "Prompt injection kind of fundamentally breaks the basic interaction paradigm ... SQL injection is child's play compared to prompt injection."
• Architectural Solutions Needed: Warns that current agent tool integrations are “built on quicksand”—security must go deeper than OAuth ([63:31]).
• Agency and Trust Boundaries: Importance of maintaining contextual integrity—AI should know when not to blend contexts across domains (e.g., “therapist” vs. “boss”) ([59:49], [60:21]).

8. The Human-AI Loop: LLM Literacy and Evolving Human Nature

• User Sophistication Matters: The value derived from LLMs is dependent on user skill—prompting is an acquired 'literacy' ([69:12]).
  • Quote (Alex, 69:12 & 70:04):
    “When I watch someone who is technically savvy, by the way, tech savviness has nothing to do with your savviness for prompting... It's a new kind of skill.”
• AI Changes What It Means to Be Human: Rather than replacing us, LLMs will shift what we can do, how we coordinate, and even how we see ourselves ([56:53], [57:41]).
• Potential for Human Flourishing—Or Infinite Distraction: LLMs could allow deeper empathy and understanding, or just infinite, addictive content ([56:53], [57:17]).
  • Quote (Alex, 56:53 & 57:17):
    “We have the potential for a dawn of a new era of human flourishing ... It could also be like infinite TV, amusing ourselves to death.”

---

Notable Quotes & Memorable Moments

• On the Path Before Us:

  “I would run onto the stage where Steve Jobs shows off the iPhone with a poster that says, ‘this will become the most important computing device on earth. It is insane to allow a single company to decide what things you may run in it.’”
  —Alex Komoroske [38:43]

• On Coactive AI:

  “It’s not you ask, it answers. You are both actively building meaning together on the same substrate.”
  —Alex Komoroske [03:03]

• On Tacit Knowledge:

  “Know-how is rich ... We used to require explanations ... now we can just move tacit knowledge between people because you can train a model with a bunch of examples.”
  —Dan Shipper [49:04]

• On Prompt Injection and AI Safety:

  "LLMs are imminently gullible and make all text effectively executable ... the combination of that and tool use is potentially explosive.”
  —Alex Komoroske [62:21]

• On Technology's Trajectory:

  “The choice isn’t preordained—you have to work for it. If you aren’t paying for your compute, it’s working for somebody else.”
  —Alex Komoroske [14:09]

• On Systemic Limits and Gravity Wells:

  “A lot of PMs are under the misunderstanding that they’re in way more control of their users and their usage than they actually are … if you’re the lead PM for the web platform, you are under no illusion that you’re in control.”
  —Alex Komoroske [41:53]

• On LLM Era's Early Stage:

  “It feels like we are halfway through the LLM era. We're in the very first innings—rubbing sticks together.”
  —Alex Komoroske [52:29]

---

Timestamps for Key Segments

• 00:00–01:16: Alex outlines the stakes of the LLM era; transformative potential vs. dystopian default.
• 03:03–05:27: Introduction to Common Tools and the “coactive fabric” concept; limitations of chatbot UIs.
• 06:46–10:14: Intentional tech, difference between intentions and revealed preferences; aligning technology with human flourishing.
• 13:05–15:35: Debate on business structures—can tech for good be built within for-profit models?
• 18:07–22:29: Open vs. closed ecosystems—AOL analogy, risk of hyper aggregation, and hope for 'web of AI'.
• 25:52–29:31: Security architectures and centralization; the same-origin paradigm and data gravity.
• 32:01–34:39: Confidential Compute and cloud security as alternative privacy solutions.
• 44:44–46:38: Alex's background in systems thinking, the ‘slime mold’ deck, organizational dynamics.
• 49:04–52:29: Tacit knowledge transfer, “liquid media”, and new reading formats with LLM-enabled AI.
• 56:53–58:19: How LLMs may redefine human potential or exacerbate passivity.
• 62:21–65:19: Prompt injection in agentic AI, architectural implications, emerging AI safety risks.
• 69:12–70:28: Importance of LLM literacy and emergent user skills.

---

Final Thoughts

Throughout the conversation, both Alex Komoroske and Dan Shipper bring a mix of enthusiasm and caution to the AI revolution. From the technical underpinnings of security architectures to the lived experience and agency of everyday users, this episode makes a strong case for building AI that serves, empowers, and truly understands its user. At its best, intentional tech offers a future where AI is not just a tool, but an extension of human intention—enabling not only productivity and convenience, but deeper flourishing.

---

For references, essays, and further AI experiments: every.to/chain-of-thought.

Transcript

[A]

We have this new technology, LLMs, that I think are as transformative as the printing press electricity in the Internet. And we have a choice. We can go down the path that we've been going down, which is engagement, maximizing hyper aggregation, you know, going after what you want, not what you want to want as a user or being aligned with our intentions. And that could lead to a new era of human flourishing. I think I intend to spend quality time with my family. I intend to experience new things. The right product fabric could enable that, but that has to be fully aligned with you. Imagine if that context all of that rich substrate of meaning that you've distilled and written down. If it's not working for you, that's terrifying. I would run onto the stage where Steve Jobs shows off the iPhone with a poster that says, this will become the most important computing device on earth. It is insane to allow a single company to decide what things you may run in it. Right. I know it feels like in this industry that we're like, halfway through the LLM era, we were in the very first inning.

[B]

It's very early.

[A]

We're like rubbing sticks together. We still think that chatbots are the main thing. Will chatbots be important? Yes. But chatbots to me feel like a feature, not a paradigm.

[B]

This podcast is supported by Google. Hey, everyone.

[A]

David here, one of the product leads for Google Gemini. If you dream it and describe it.

[B]

VO3 in Gemini can help you bring.

[A]

It to life as a video now with incredible sound effects, background noise and even dialogue.

[B]

Try it with a Google AI Pro.

[A]

Plan or get the highest access with the Ultra plan. Sign up at Gemini Google to get.

[B]

Started and show us what you create. Alex, welcome to the show.

[A]

Thanks for having me.

[B]

So, for people who don't know you, you are the co founder and CEO of Common Tools, which you'll describe for us. You were also previously the head of corporate strategy at Stripe and director of product management at Google. And I met you, I think, like six months ago. And the thing that really stood out in our conversation, aside from just like you have so many interesting ideas, is you're this really interesting systems builder. You think a lot about personal systems work systems, things that help help you get more done or think differently about the world. And I think people like you who have that kind of like, obsessive mind with systems, is something that's something I recognize in myself and, and also I think are people who have a lot of interesting things to say about just new AI ways, because I think it's sort of like steroids or catnip or whatever you want to say.

[A]

Some kind of accelerant. For sure.

[B]

Yeah. For people like us. I'm excited to chat with you and excited to hear both what you're thinking about now, what you're building and what kinds of systems you're playing around with personally.

[A]

Yeah.

[B]

Cool. So why don't we start with common tools? Because I think there's something interesting about what you're building because I think it speaks to the perspective you have, probably about this larger AI wave. Tell us about it.

[A]

Yeah, the way I think about it, and it's hard to describe because we're trying to build a new kind of thing that's only possible in the world of AI. So when I describe it, people say, oh, you're being coy. It's like, no, I'm telling you what I'm building. It's just. It's a new kind of thing. It's hard to grab onto. I think of it like a coactive fabric for your digital life. And by coactive, that's an old word that I hadn't heard before, but I think is a perfect fit for it. Of you are active in the system and so is this emergent intelligent process, your private intelligence, powered by LLMs, and those can be coactive on the same substrate and adding detail or adding the little bits of software or connecting up things. And I think that that really unpacks or unlocks the power of LLMs.

[B]

I love that word. And it's interesting because I think we're finding something similar for some of the products we're anchored in Internally is like, right now, the paradigm is you send something to ChatGPT and it sends something back. Maybe now it sends something back after 10 minutes, which is a new thing. But it's not like you're not working in parallel, it's not working in the background. And then being like, I had this idea and we're starting to, as we build our own sort of agentic systems inside of every four, during your email or for writing or whatever, we're finding that having something that's working while you work with you and, and, and being a little bit more proactive is actually a really interesting next paradigm.

[A]

Yeah, to me, I mean, if you, if you say, okay, we have LLMs, what's the first product? In 10 seconds you go, chatbots. It's the most obvious extension and everybody is focusing only on chatbot. Like, will chatbots be important? Yes. But chatbots to me feel like a feature, not a paradigm. And they are so like, they're, they're like this coactive sur surface but it's just append only I append a message, it appends a message, I append the message, it appends a message and literally back and forth. It doesn't have a way to like have multiple for me in a row. And you know, chatbots are amazing for starting any task because there's no structure. You just say whatever you're thinking and it helps to still and respond. But for long lived tasks you need structure. You want there to be something where you can glance over and you know, a little cubby hole where you put certain information. I find I have, you know, I do dozens and dozens and dozens of chats with ChatGPT and Quad A day and I'm just swimming in all that context and all the, all the chats and so I'm using it to do important things and also I'm just completely lost in it because it's only this chatbot kind of penned only kind of paradigm.

[B]

You have this really cool thing called bits and Bobs which is this like real time Google Doc that you just have been updating for I think like many years at this point with things that you're thinking about, which we'll link to in the show notes and I love, I think it's, I think it' really cool and really inspiring. Just sort of the rawness of that. And one of the things that's in that Google Doc right now about what you're thinking about is this idea of intentional tech which dovetails really nicely with something that I've been thinking about, which is this idea that AI has. This is this new era of technology where technology can understand our stated preference, stated preferences, which is different from the social media era, which worked totally just on review, what you dwelled on, what you clicked on, whatever. And so social media tends to make us, tends to serve us things that are like, like more outrage driven or more like sexier or whatever. Anything that catches your eye. Yeah, like the car crash effect basically. And AI, because it can talk to us, gets a much richer understanding of who we are. Maybe like I think ChatGPT is much more helpful and much more enlightening for example, than like the Facebook algorithm is just by default. So I think that dovetails a lot with what you've been thinking about with intentional text. So tell us, tell us what's on your mind about it.

[A]

Yeah, intentional tech I think is really critical in this new era. We're at a crossroads as a Technology industry. I think we have this new technology, LLMs that I think are as transformative as the printing press electricity in the Internet. So it's this big general purpose unlock for all kinds of stuff that wasn't possible before. And we have a choice. We can go down the path that we've been going down, which is engagement, maximizing hyper aggregation, you know, going after what you want, not what you want to want as a user or being aligned with our intentions. And that could lead to a new era of human flourishing. I think this is not the default path, by the way. We will have to choose it. We'll have to work to build that. And that's why I think intentional tech is so important, because we want technology that aligns with our intentions. Not necessarily what I, you know, what I want and my revealed preference of like looking at the car crashes or whatever, but what I intend to do, I intend to spend quality time with my family. I intend to experience new things. Um, I. I intend to read interesting takes that disagree that give me disconfirming evidence that challenge my worldviews. Those are some of the things I intend. That's what I find meaningful to do. And it's very easy to fall out of that. But LLMs, I think with the right product fabric could enable that. And another important part of this is it aligns with your intentions. As in, it is working for me is an extension of me. Um, I heard a word last week I love of Exocortex. It's like an extent you're caught in that exoskeleton, but that has to be fully aligned with you. Imagine if that context, all of that rich substrate of me, meaning that you've distilled and written down, that's all about, you know, the things you care about and all these facts about you. If it's not working for you, that's terrifying. Like, I saw someone a few weeks ago at one of the big tech companies, said, we're making our tool personal, proactive, powerful. It's like, well, let me stop you right there, because the very first word personal doesn't actually align because you are a massive corporation that's trying to sell me ads. And if you're maintaining a dossier on me and then that dossier is leading to a powerful and proactive thing, that's terrifying. Whereas, if it's working just for me of the extension of my direct agency, that's really empowering. And I think that's one of the reasons intentional tech is so important at this era.

[B]

That's really interesting. And tell me more about like, because, you know, when I think about this, one of the, one of the words that comes up you're talking about alignment is that AI started or this generation of AI started with alignment or alignment with human preferences at, at its core, because we were all afraid that AI was going to kill us. And now it's like, well, it might not kill us, but maybe it'll just serve us ads that, that will make us dumber, spend our money in ways that we shouldn't or whatever. So I'm kind of curious what you think has to happen to make the, make us go on this sort of more intentional path.

[A]

To me it's alignment. I think it's funny. Yeah, we use it for the underlying LLM model. I think there's two layers. There's the model itself, which is if you imagine this as a stateless thing that you send a query to and it gives you back a response, it doesn't store any state. Then you have alignment problems of what kind of biases are baked into it and what have you. But then there's a layer on top which is your context and that is the thing that is a malleable bit of information about you that changes. It has lots of rich meaning on it. These are two separate layers, I think. And so I am more interested in this layer at the top. And assuming if you have multiple LLMs that you can choose and swap between that don't remember anything about you, then it's fine, you can swap between different ones. And that matters less to me. Where context is the place that's all aggregated is I think, more important.

[B]

Well, that's an interesting architectural question. Like right now, language models are stateless. Do you think, like in five years, for example, there's still going to be these sort of stateless intelligence boxes? Or do you think they're going to be auto updating their weights, for example, as you talk to them?

[A]

I, I don't know about auto updating their weights. And this might be. There's an architectural breakthrough, but this is one of the things that's weird about these models is that they do take a long time to bake. So like they have this weird paste layer down at the bottom that's like months behind on training data and it just takes time for it to bake. And then you have these like, you know, layers on top that the system prompt now, like in ChatGPT will inject little bits of your context, a weird bit that, by the way, you cannot inspect which is kind of funky, right? Like if you ask it, hey, what do you, what context about me did you put in there? It's like I can't tell you. Like that's creepy, right? I can't do that. Dave, you know, and Simon Willison, we were chatting last week and you know, he's got some prompt that will extract out this dossier about you. It includes things about stuff you said in the past. It could include things like you're insecure about your weight, you know, these things about you.

[B]

Yeah.

[A]

And then it also says in the last, in the last, you know, five weeks, 9% of your interactions with the chatbot have been bad. As in you've been trying to manipulate it or whatever. Like, I don't know, it's weird seeing this, this, this view into this for yourself. So I think ChatGPT and others are very clearly trying to move, merge these together so that you have all your contacts in ChatGPT and it's hard to leave that. I think it's more problematic especially if you're going down an engagement maximizing kind of playbook, which I mean a bunch of, you know, the, the executives from Facebook now run three of the four big chatbots. You know, I think globacus depth. To me, full computing has four major components to me. One is it's human centered, not corporation centered. It's centered around me and my intentions. Two is it is private by design. That means that the data is only for me to see. It's for me to bring where I want to bring and to choose who gets to see it using techniques like confidential computing to make sure it is entirely private. It has to be pro social. It has to be something that helps you live a life integrated with society, not just being this hyper individualized kind of, you know, little islands yourself, but integrate with society in a meaningful way. And the last piece, which I'm going to remember off my head is it also is open ended. It has to be something that allows you to explore and to build and to create new experiences within it. The way that our software works today is only the, the entity that created the software is allowed to decide what kinds of things you can do in the software. And that's due to the security model of how, of how our, all of our software has worked for the last 30 years. But that kind of close endedness means that like you had to convince some PM somewhere to have prioritized your feature, which if it's particularly niche, probably hasn't been, probably hasn't happened.

[B]

Okay. Well, there's, there's a lot of things you just said. I would love, I would love to dig into it, dig into all those points. But the big one that stands out to me is right now we live in a world where in order to get this great new technology that's advancing at this really rapid pace, we have to get it from well funded startups that have a profit motive. And I know OpenAI is a nonprofit, but like, you know, whatever.

[A]

I think that's what they say too. Yeah, whatever.

[B]

Clearly they're being run like a, like a startup. And I guess for me, like, I don't think that's a bad thing, but it sounds like at least one way to read what you're saying is we have to get rid of the sort of like corporate startup structure for these technologies to be not necessarily a social way. Like how do you envision this being made, us being able to make these decisions in terms of what kinds of corporate structures or startups or nonprofits or whatever are controlling and building these things to get the outcome that you're talking about?

[A]

I think it is compatible with being a business. So for example, we're charted as a public benefit corporation. Um, the, to me it is what's most important is that users are paying for their. Like it's not free. Um, if you aren't paying for your compute and it's not working for you, it's working for somebody else that just so happens to think that giving it to you is a benefit. Now that doesn't mean that's a necessary but not sufficient characteristic because I think ChatGPT's the playbook that they're executing is one that is about engagement maximizing and making you stickier to the service and what have you. But you paying for your own compute is a necessary component of it to make sure that it's actually truly just working for you. That's where also that private privacy really matters of your data is visible only to you with your keys. This does not necessarily mean by the way people have historically in the past said, oh, that means it has to be local first. I love the Local first movement. I'm very aligned with its ideals and ethos. Local first is really hard for a couple of reasons. One, architecturally it's very challenging to have a bunch of peers that have to have a visual consistency. You don't know when they're going to sync back up. It's very hard to build consumer product experiences that work the way that people expect. And two is it's Inconvenient if you got to run your local server and your laptop isn't plugged into this, into the network when you're on your phone or whatever, the thing doesn't work. So actually there are other architectures that can use things like confidential compute to run in the cloud in a way that is remote, can be remotely attested to be totally private to you, and that nobody else can see into that data. And so those are some of the architectures that allow full alignment with the humans.

[B]

Okay, so Basically you're saying ChatGPT is, ChatGPT is doing some of this. So they're, they're, they're starting with a subscription model. But there are things that they're doing, like, what are they doing that you think is like, aimed at maximizing engagement? I definitely understand the kind of like, memory thing. I want to keep using ChatGPT because it remembers who I am. Like, I love that part of it. What are the things that they're doing and the decisions that you're, that they're making that tell you that they're, they're kind of going down this engagement maximizing path that will lead to a bad place.

[A]

I think it's the engagement maximizing path is just the default path. I don't think, by the way, there's anything particularly. They're like, oh, here's our dastardly plan. This is how you do tech now. Right. You're like, in the last 10 years, we just realized that aggregators are a really powerful business.

[B]

Yeah.

[A]

And they want to consume all of the demand, so then all the supply comes to them. And that's a great business. I mean, if I were sitting at their seat, I can totally see why they would do it. Like, that's again, it's not a nefarious plan. It's just the default thing that you would do.

[B]

Yeah.

[A]

I think if AI is this incredibly important technology that also knows all the intimate details of your life, that becomes especially important for it to be in an architecture that is not something that you're. Something that this other company can look at and maintain a dossier about you. Because that it's just too easy to manipulate. Right. Like, LLMs can, can translate anything from anything. And that means they can also translate just for you to figure out exactly how to land a particular message for you. Which means it's imperative that you know it's aligned with your intentions and it's not going to try to like, get you to do the, the, you know, the Mondo subscription or to go to one of the partners or what have you.

[B]

Well, let's say you're, you're Sam Altman right now and you're in charge, you're in charge of OpenAI. What would be, what are the decisions that you would make tomorrow that would put you on a path that you think is actually right?

[A]

So if I were Sam Altman, I would do what they're doing honestly. And as in it's, I can see why it's a great business. I think we'll add a lot of value. That's why I see this as a complimentary approach to some degree of making a system that is. This coactive system that's just working for you, that is not just chat can be a complementary thing to these other systems. I wouldn't say oh, you should change what you're doing Sam Altman, because I think actually what they're doing is a reasonable thing. These models that they're creating, especially because they're available via the API with no memory stored to them, that's an important characteristic, is great and it makes a really powerful underlying engine that can be used to power lots of other products, especially to the extent that there's multiple that we can swap between and don't get stuck.

[B]

Got it. So you think that there's a call for a complimentary technology, technology provider, set of technology providers that allow you to take all of the context that you have, keep it private and then bring it to any situation or service that you want.

[A]

Yeah, and I, I would say the, my mental model is ChatGPT is kind of the AOL of this era. So if you use the parallel of the Internet and the web, AOL was an extremely important company. They're the ones that brought everybody online. America, the whole of America Online has amazing experience. It was somewhat of a Wal Garden, but it also gave you access to the, to the open endedness of the open web. And I think that's the kind of role that I, I see them playing. I hope that what we get out of this is not some closed ecosystem of like hyper aggregation beyond even what we've seen to date, but instead we see this open ecosystem of things that are very user aligned.

[B]

Interesting. Why AOL versus like Microsoft for example. So Microsoft, I think in a similar way, not for the Internet era, but for the PC era, like taught people how to use computers and brought computers as a thing to most of America and the world and built a platform that everyone else is sort of building on top of. So that's another kind of analogy. Why do you Think AOL is more appropriate.

[A]

I think AOL, the analogy that tracks me best for LLMs is the Internet. Like LLMs are the Internet. They're a new kind of thing that you can do aol. At the beginning, when I first started using it in like second grade or whatever, it was about chat rooms and you could do the, whatever the keywords or whatever. Yeah, asl. And that was what I thought the Internet was. And then it turns out, as I learned later, I was like, whoa, this is a whole crazy open ecosystem that no one controls with all kinds of weird happening. And that was that open system of the web. And so in the same way I see them as distributing this new technology to people showing, yeah, look, chatbots, you can do these cool things with chatbots. And then later you realize, oh wait a second, the same thing that animates chatbots could be used to animate all kinds of new experiences I didn't realize were possible before. And that's one of the reasons I think that them as the AOL versus a Microsoft.

[B]

Why did AOL die? Like I don't remember.

[A]

I think at a certain point the, the open endedness of the web could have took over and it just became the point of like, like I think a lot about systems. As you pointed out, I think open ended systems tend to win under certain conditions, especially in the growth era. Like at the very beginning there's some vertically integrated thing that kicks off a new revolution. People go, oh, it packages up really nicely and people get it. And as time goes on and it gets the sort of combinatorial possibility gets too big for any one entity to successfully execute. And so the open system, if there's any way for it to escape out in the side like it was with the web. Sorry, I talk with my hands as you can see if there's any way for it to sort of escape out the side like there is with the web or using APIs to access LLMs, then the open system kind of takes off in this combinatorial swarm and it overwhelms the closed system. That's also potentially just a hopeful thing. I hope. We typically see oscillation between open systems and closed systems. We've been in closed systems dominant for like the last 10 years. And I think it's a bummer. Like I think we're kind of in this, we're in this amazing age of technology and it's also kind of the dark ages of tech because we're all in this hyper aggregated thing where the only features that can exist are the ones that aggregators for consumers are the ones that aggregators have decided to prioritize. And that's a bummer because as a company gets larger and larger and larger, the Cosian floor, what Clay Cherokee would call the Kosian floor, the smallest feature that it makes sense for them to even consider prioritizing goes up and up and up and up and up. So if you're at Google, if I'm trying to pitch a product to other teams and they say, how many net new one day actives do you think this product will get? I'll say, oof, I don't know, 50,000? And I go, 50,000? I wouldn't do this for 50 million, you know, so like there's this whole class of features that just can't exist in a world, the world that we're currently in, that's hyper agated.

[B]

It seems like one big difference between the web, the early stages of the Internet, which to be honest, I don't know the history like that well off the top of my head, and language models is the web was like you have, I mean it was sort of government run at first. You have this like international consortium that kind of just defines the standards for how the web works and all that kind of stuff. That doesn't seem to be the case here.

[A]

So I don't think, actually, I think we are waiting for the web of AI to show up.

[B]

Interesting.

[A]

And that's partially what we're trying to help catalyze. And I think that's what I mean by we have the Internet and we're like, cool, Internet's definitely going to be useful. Like if you look back at Al Gore's proposals for the information superhighway, it was actually very prescient in a lot of ways. It's just, it was all about pipelining existing content and business models into people's homes. And some of that definitely happens. You get things like Netflix or whatever, but a lot of like YouTube, social networks. None of that was envisioned by the, or the, you know, Wikipedia. None of that was envisioned back then because we just didn't know how to imagine it. It wasn't even the thing that we thought was possible. And then the web creates this open system of lots of different people trying out different things and seeing what things work. Found some of these interesting new pockets of value that grew into whole continents of value.

[B]

So how do you, how do you make that though? Like, do you have to, is it, do you have to get governments involved? Do you have to, is it blockchain where everyone's like sort of running.

[A]

Oh no, no, that's what it sounds like to me. Can build. We have so many powerful substrates. The web exists as a distribution platform. Everyone's, everyone's device speaks to the web and so you can use it as a way to distribute a whole new kind of experience that fits within this. So I don't think back in for the Internet you needed a consortium, you needed tons of capital expenditure to build the pipes and that was government grants. It was also businesses that way overbuilt all this capacity. Those businesses by the way, were great for society. We got all this excess bandwidth and capacity. Kind of crappy businesses though, right? Because you build a pipe somewhere, if someone builds another one right next to you, you lose all pricing power. You are just a total commodity. But it's a commodity that powers the rest of this innovation on top. I kind of see large language models, the producers of them as people laying the pipes. They're the ones doing this extremely capital intensive creation. But there's actually not particularly that much of an edge across them, which is great for society because that means you can take LLMs as a commodity, you can assume that they exist and reasonably high quality ones, including quite good open source ones, and that means that all kinds of interesting business dynamics take off and you can kind of take it for granted and build what's up here. Now that I can take high quality LLMs for granted.

[B]

Yeah. It does strike me that the thing you're, you're talking about, which is the bar for like a Google PM to build a feature, is like we need 50 million users. And so you sort of, you sort of look past the like 50,000 user use case which A, those are just valuable in themselves and B, often they're the ones that end up being the 50 million user use cases.

[A]

Some subset of them will become like.

[B]

The sort of innovator's dilemma type stuff.

[A]

Yeah.

[B]

It sort of strikes me that in a world where language models are commodities, capitalism just does that automatically because small startups will just like do the 50,000 user use case and just, and just build it. So is this just going to happen?

[A]

Here's what it might and I agree that like this kind of. I'm so glad there's so many different ways that the society could have gone OpenAI could have had ChatGPT before they released the API. That would have been such a different world because now everyone assumes they have to compete with an open access API and now it'll be very hard to close that door, thank God. So we can take it for granted for the rest of us. So, like, I just. I, like, woke up with a, like, last night with a nightmare that, like, it had gone the other way, you know, so I'm very glad.

[B]

Or like, IBM developed AI.

[A]

Yeah. Yeah.

[B]

Oh, my God. What would, like.

[A]

How terrible.

[B]

$30 million a year for like, one query.

[A]

Yeah, exactly. So that's great. Here's the thing, though. That means that the current laws of physics, the current security model that we use for the web and apps, actually limits this possibility. And this security model is called the same origin paradigm. The only people who know about it, really, even web developers, don't really know about it.

[B]

I have no idea what you're talking about.

[A]

What is that same origin paradigm is the laws of physics, of how the web works. And what it says is it's the security and privacy model that we've used for, since. For 30 years. It kind of actually grew up as an accident, actually. And at the very beginning, the web has no state. And so you reach out to a server, it gives you back the same thing that anybody would have gotten if they gave that exact same request. Well, then you add cookies. And then you need to say, when do I send. Where do I send cookies to once they've been set? Which other URLs do they go to? And then the easiest thing is, okay, there's a notion of a site which is roughly a domain.

[B]

I see.

[A]

And so it will send things back and isolate them by that. So cookies will come back between these, but not go over there. And then you add JavaScript and you allow things like local storage in local state. Where does that look? Who can see that? Local estate. The same people that can see this. And so now it grows up as this or as this origin boundary. So this somewhat happenstance thing turns out to be at the core of their entire security model. What it assumes is each and every origin is its own isolated little island. It can't see data from any of the other origins, but it can see all the data that the user put into this origin, intentionally or unintentionally.

[B]

An origin is roughly like google.com is.

[A]

Roughly, roughly a domain. It's not exactly, it's like slightly different, but it's roughly, you can think of as a domain.

[B]

Yeah.

[A]

And so what this means is data accumulates inside that origin as a little island. This is a very reasonable and good security model. It's one of the things that makes clicking a link or installing a new app safe, because it knows Nothing about you. So then if you choose to like it and to put more data into it, that's okay, but that's your prerogative. So it's great for trying out a new thing. The problem is putting new data into it. It can do whatever the hell it wants with that data. It can send it to evil.com or scammy.com and you have no idea. You're implicitly trusting the creators of that, of that application. And so somewhat surprisingly, this leads to massive centralization. So if you have use case. So this, this model is about isolation, not integration. But our lives are integrated. And so if you want to move things across different origins, you have to be the orchestrator. You as the human have to keep track of the information you want to copy, paste or move between these different things. It's expensive and it's somewhat scary to put it into a new thing. And so imagine that you have two use cases. One is some cool startup that says they're going to do this amazing new thing with your Calendar and scheduling and one is Google Calendar that says they have a similar kind of feature. Which one do you pick? Well, Google already has all your data in this startup. You don't know if they're securing the data properly or what the business model is.

[B]

Plus you have to give it the data like step by step, like it doesn't it.

[A]

It doesn't know it to start. It's a cold start problem. And so this leads to ask, I'll just do the one that already has it. So this is a phenomenon you might call data gravity. It tends to accumulate in the places that already exist and they become massive. And nobody else can get this data. And this is one of the drivers of aggregation. And again, once you aggregate to such a scale, your cosine floor goes up and the set of features that they could even consider doing is only a small subset of what you could do with all of this data. In practice. This all arises somewhat surprisingly from the security and privacy model that we built.

[B]

I love that. Okay. And so, and I think what you're saying is because we inherited that from the first generation of the web or second generation of the web or whatever chatbots currently operate that way too.

[A]

We don't even realize the apps all do this technically. Legacy applications on desktop do not have this model because they can interact via the file system. Apple has been retconning and jamming it into Mac OS X for a number of years. Windows probably has to admitting attention. So people, when we're building software and operating systems. We don't know another way to think. It's like, how else would you possibly do it? Which is insane. So there's this. I call this the iron triangle of the same origin paradigm. There's three things you can only have. Two. One, untrusted code, two, sensitive data. Three, network access. You can only have two of the three to have a safe system.

[B]

I don't understand.

[A]

If you have untrusted code with sensitive data and network access, it comes in, it looks at your thing, it figures out your financial login and sends it to easily.

[B]

So untrusted code is like some developer wrote some app that I'm downloading that hasn't been. I don't look at. Or in what sense is it untrusted?

[A]

Untrusted as an. I haven't made a trust decision about it. So when you put an app in the App Store, Apple looks at it and says, okay, based on the construction of the sandbox and also based on our review of this, this is fake. This is fine, and we'll allow people to install it.

[B]

Okay, so untrusted code is first thing. What's the second thing?

[A]

The second thing is sensitive data.

[B]

Sensitive data? Yeah, yeah, you don't want to potentially.

[A]

Identifying data, potentially precious data. And the third is network access. So the. The web says you can get untrusted code, web pages and you can get network access, but no sensitive data. You get only the thing. Exactly this. The app model says you get sensitive data, network access, but not untrusted code. It all has to go through the central location. That, by the way, starts as a 30% tax.

[B]

What about, like, Windows? Because Windows doesn't have an app store. So what's the.

[A]

So historically the model, there wasn't one. And that's why installing software and those is a little bit more dangerous. Because without a model like this, it could do whatever it wants and it has access to all that data. And you have to be more careful.

[B]

But it still exists. Like, it still exists. It's still like a vibrant ecosystem. So it's possible to do it without this triangle.

[A]

And you could also change it. You could tweak it in other ways and you could do clever things about using tools like information flow control and confidential computing to create a whole new sort of laws of physics, I think. Okay, so and if you did, then a whole. You could do things that hit all three. If you had all three, you could do wild, interesting things that would not be possible today.

[B]

So is. Is the file system your number one, like, example of like A good alternative to a same origin paradigm.

[A]

Yeah. So the file system allows fundamentally multiple apps are all allowed to work on the same data and they can coordinate via the file system.

[B]

Right.

[A]

And that allows you to do. To not get stuck. It's. When you think about it, it's insane that all of your data is locked up inside of an app. Like it can't leave the app. That's wild actually when you think about it.

[B]

Well, it's interesting because the file system has a number of properties, unless you don't sell Dropbox or whatever. But like it's here, my file system is here as opposed to like it's everywhere. And I kind of have physical and just total control over what happens to it. Is that an important property of the.

[A]

System, you having physical control of it? That's why Local first talks about, oh, we're going to make Local first where you have control and you can make to bring the data across different things. Part of the challenge again with Local first is it's inconvenient. We expect things to work across multiple devices to work even when one device is off. We expect these things to be in sync across different things. And the Local first architecture is quite difficult. This is one of the things that we are looking at we call open attested runtimes. The pattern is use Confidential Compute. So Confidential Compute is secure enclaves in the cloud. It allows VMs to be run fully encrypted in memory, which means that even someone with physical access to the machine, like a Google SRE can't peek inside, which is great. Then what you do is you have an open source runtime that is you can that you run, that executes the code. And then you can ask the Confidential Compute cluster to do a remote attestation and to give a attestation assigned attestation by the underlying hardware manufacturer that says this thing that you're talking to that just handed you this is running in Confidential Compute mode. And here is the SHA of the git sha of the VM that it booted with. And so it allows you remotely as a savvy user to verify, oh, it's running an unmodified version of that software. And that solves a big coordination problem because now lots of other people could all verify that it's running unmodified version of the software. And yes, fine, we will just use that central server as the place to coordinate because we can all see it can't do anything different than what it says it will do. And that's a really powerful coordination do.

[B]

Users care about that? So, because one of the things that makes me think of is the. A lot of these concerns feel similar to the flavor of concerns that like original blockchain crypto type people had and were solving for with, with Bitcoin or, you know, any, any kind of, any kind of cryptocurrency. And then Coinbase came along and they were just like, we're just going to put a general, like, sort of same origin paradigm type solution on top of that. And people actually love to use that. So is this, this is evidence.

[A]

I think if you go after users who care about this as a primary end, you end up with a very small audience of people and you end up with a mastodon, for example, as in something that's like pure but also kind of finicky to use and doesn't make that much sense. And that's why I think you want a blue sky kind of approach. Blue sky you can use as an end consumer and be like, the only thing I know about it is it's that, you know, it's not owned by, you know, that, that guy, you know, and that's it. But then the closer you look, the more you realize, oh, this is actually very clever. This is an interesting way of, like my key pairs and the way that personal data stores work. Most users will never have to know about that, but the more you learn, the more you're like, oh, okay. And so that I think is a, what I would call like an inductively knowable system.

[B]

Yeah.

[A]

People go, I was talking to somebody like, nobody know. Nobody ever cares about the security model. They do actually care about the security model. They just don't know the words to express it. Nobody understands what the same origin paradigm is. Nobody. A very small portion of people. And yet it is the laws of physics that make all the other stuff we do safe. It's the reason you don't have to care is because some, some also do care. And the general characteristic for me is when you're using a new system, you're like, ooh, this seems creepy or too powerful. And you go talk to your more tech savvy friend and you say, hey, Sarah, do you trust this kind of thing? And she says, yes, I do. And the reason I trust it is because. And she knows. She's read a blog post on Hacker News that was someone who wrote read a thing, someone who audited the code. And that inductive change goes all the way back down to the fundamentals.

[B]

This episode is brought to you by Addeo. The AI native CRM built for the next era of companies with Addio setup takes minutes. Connect your email and calendar and it instantly builds a CRM that mirrors your business with every contact enriched and organized from the start. From there, Addio's AI goes to work. It gives you real time intelligence during calls, IT prospects, leads with research agents, and it automates your team's most complex workflows. Industry leaders like Union Square Ventures, Flatfile and Modal are already building the future of customer relationships on Adeo. Go to addio.com every and get 15% off your first year. That's a T T I O.com every and now back to the show. That makes total sense. I totally agree with all of that. I think the, the thing on my mind is the first people that are going to adopt this probably are the people that do care about the security of it. Right? And are those people the right people to kind of seed the community that, that ends up blossoming into this thing? Because those, that kind of person is going to say a lot about like the, the total trajectory of this kind of product.

[A]

I actually, I don't necessarily agree that it will be those people, the privacy heads that are adopting it earlier. I think of it more as like the high volition users, just the tinkerers, the people who are the early adopters. Some of them will have a higher proportion of caring about this kind of thing, but a lot of them won't. Like there's a ton of startups that they, they say step, you install the app, step one sync your Gmail inbox and boom.

[B]

Okay, I, I run a startup like that.

[A]

Exactly. Because it's. Most people don't really care, especially early adopters. Yeah, I think what we're describing is a system that if it, if you can break this iron triangle and then you get these all these crazy emergent phenomena that aren't possible in other software. People use it because of that and the reason it's not creepy is because of an underlying security or privacy model.

[B]

Do you have a name for the alternative to the same origin paradigm?

[A]

I call it contextual flow control.

[B]

Contextual flow control. Tell me what that means.

[A]

I don't want to go into too much depth at this point. Still very fuzzy. Helen Niesenbaum is a professor, I believe at Cornell, a legal studies professor. And she talks about contextual integrity. Contextual integrity is the gold standard of what people mean when they say privacy. So when people say, when lawyers talk about privacy, they think about consents. Did the user sign a Consent that said we can do this. That's it. Signed the eula. It's fine. Technologists talk about end to end encryption. As long as it's end to end encrypted, it is private. But what people mean intuitively is this contextual integrity that the data is used in the context that you understand and align with your interests. It's not surprising how it is used and it's not going against what you want to happen. And that's like a sort of first principles, ground truth way of thinking about it. And then you combine that with things other technologies and allows you to make formal claims about information flows in alignment with people.

[B]

That's interesting. The re. The reason I love the same origin paradigm thing is like you're going all the way back to this like one decision that like has all of these really interesting positive and negative effects like later that are sort of unpredictable, which feels very, it's like a little bit Steven Johnson or a little bit like, like one of those writers that's just like, like here's this one thing about the way the history works that like just totally changed everything.

[A]

Yeah.

[B]

Do you have any other things like that that you've noticed about technology? I don't want to put you on the spot, but I just feel like you probably have some, some sort of counterintuitive things in, in your time working at Stripe and Google and whatever, thinking about like mobile or you know, SaaS or whatever.

[A]

I think a couple other like the world changed that day. We didn't realize. One is the ChatGPT coming out after the API was already available. Another one is if I could go back in time, like at a time machine, I would go to the stage on 2007, the stage where Steve Jobs shows off the iPhone and I'd run onto the stage with a poster that says this will become the most important computing device on earth. It is insane to allow a single company to decide what things you may run in it. Right. Like that's insane. That's absolutely insane. And the only reason it's actually viable is because they decided very shrewdly at the very beginning to not allow other rendering engines in browsers on that device from the very beginning or and to make sure that all software had to go through the App Store. And that was a reasonable decision. And like the first order implications are very reasonable. It means that you get as a user this safe experience. You know, nothing's going to hurt you in it. The second order implications of that are wild. Like it's, it's insane to me that like the most important computing devices in our lives, a company who has demonstrated again and again that they are willing to use capricious kind of decision making about what things will distribute is the one that like that gatekeeper. That's insane to me.

[B]

It seems kind of like a lot of what you're reacting to in, in the Apple example and the same origin paradigm example is aggregation of power and a few large big tech companies and the kind of second and third order negative effects of that on user experience and innovation and stuff like that.

[A]

And competition and.

[B]

Yeah, competition, yeah. So what are the. It seems like that's a reasonable place also for like regulation. What is your stance on regulation?

[A]

And regulation also has all kinds of second and third order impacts, you know. And so have you ever met Danelle Meadows thinking in systems? Yeah. So like the systems you gotta dance with to some degree, you can't fully control them, you can develop it. And that's why I find some of these characteristics when you find the right like technical leverage points you can say oh actually if you go. That has a very different way that it evolves. I think is, is preferable to a regular regulatory.

[B]

What if I said like you could pass, you get your president for the day and you can. And, and, and all of Congress.

[A]

You'Re.

[B]

A dictator for the day. Yeah, yeah. And you can pass one law. Do you have like what, what law would you pass? It's like a, you can, it's a one sentence law.

[A]

I don't know know. I would have to. I, I don't, I've never thought through like I, I'm so used to in open systems which we're all part of a large complex adaptive system called society and there's always like weird eddy currents of power dynamics and what have you. So like I take from the assumption that like I never have a lever that is like this massive lever I can just simply pull and so I don't even know.

[B]

I'm more like looking. You want to find the like grain of sand that you, you put on the pile and it just all like cascades.

[A]

But it cascades if the pile is ready for it.

[B]

Right, exactly.

[A]

And that's one of the nice things about like systems thinking is like if you do the right like judo moves the right, right thing and the system is ready for it, the world moves and if it's not, it doesn't move. And so the system is deciding kind of through all these like micro interactions throughout it.

[B]

Tell me more about that mindset and how you developed it, and what are some of the key moments in your life where that you've seen that happen?

[A]

So I wrote my undergraduate thesis on the emergent power dynamics of Wikipedia's user community, and my degrees in social studies with a minor in computer science. And I was just fascinated by this emergent phenomenon of how always a stranger work on this thing with no coordination, and yet you get this convergent, extremely important result out of it. That's insane. It's wild. And then when I went right out of college to Google, it actually was harder to get my first PM role because I didn't technically have a CS degree, even though I basically did, it was one credit shy of a dual major. And so I was like, man, what a mistake that was. Like, I just didn't just throw it in the back of my mind like, whatever, if I could go back in time, I just would have majored in cs. And then I did my first year on search, the precursor to the knowledge graph. My second year in DoubleClick, I was in the APM program. And part of that is you mentor a lot of the people who come up after you. And I learned very early on I loved mentoring people. It helped me think through what I was dealing with and help people and get more patterns to go in pattern matching library. And then I became the lead PM for Chrome's web platform team. And I think a lot of PMs are under the misunderstanding that they're in way more control of their users and their usage than they actually are. If you're at Google and you ship a feature and tomorrow 50 million people use it, you kind of get a little bit of a God complex, right? If you're a platform pm, you tweak a thing which causes other developers to do something different, which causes users to be affected in a second order effect. You're aware of this indirection. If you're the lead PM for the web platform, which is a open system with multiple browser vendors who don't like each other very much and are constantly kicking each other under the table, you are under no illusion that you're in control. And my engineering counterpart, Dmitry Glazkov, who was the Ubertl for Blink, brilliant guy. And he was the one who got me into, introduced me to the word complexity, for example, and introduced me to the Santa Fe Institute. And so as I was going, I was realizing, oh, know these things I'm naturally doing to try to make these good outcomes happen in this open ecosystem like progressive web apps and these web standards and stuff. I'm intuitively applying some of these power dynamics and complex systems. And then I left Chrome, I went to go work on augmented reality. I created a little nerd club behind the scenes with a bunch of people that I just kind of selected that people who, when I said it was a nerd club, didn't go, ew, like, you know, like ah, that sounds fun, Great, come on in. And then it was very collaborative debate, very, you know, trickling in different perspectives where I get diversity perspectives into the system. No particular goals it's exploring. After a while we came up with a distilled like a thing. I was like, wait a second, this must be the strategy. Like nothing else could work. This is a strategy that kind of makes sense of all the pieces. And that caught on a significant amount of momentum. And that's when I realized, oh my gosh, I'm not a web developer ecosystem guy, I'm a systems guy. And the same techniques I've been advising I used and deployed in those contexts are the same techniques I used. I've been advising, you know, hundreds of PMs over the last decade to use to navigate Google. And the reason they work, the lenses I'm using about game theory and power dynamics and evolutionary biology are things I learned in college. And it was this kind of like aha moment. And that's when I wrote the earlier version of the slime mold deck.

[B]

Tell us, tell us about that.

[A]

That slime mold deck was a kind of lightning in a bottle. It was, it's a, I think it's 150 slide emoji flipbook. And it's just about a, a fundamental characteristic of system organizations that as organizations get larger, they get much, much, much slower. And that's true even if you assume everybody is actively good at what they do, actively hard working and actively collaborative, and it arises due to a exponential coordination cost blow up. It's fundamental, it's inescapable and it's the force that we all deal with constantly and we don't even realize we're dealing with it. And so we get frustrated with ourselves, we get frustrated at Jeff over there who if only he would do X, this whole thing would be easier. And we're just all fighting with this massive force, this force of gravity that's completely invisible to us. And so that deck caught on. I did another version, similar kind of thing externally. And that also got a surprising amount of momentum. I've had people describe it to me as like life changing. I'm like, I'm just talking about like some emojis talking about game theory. But it was. I think it really tapped into something that people experience, they're frustrated by, and they don't just. It's like a big hug that says, you are not crazy. This is really hard. And it's fundamentally hard.

[B]

So I think like every. Every nerd that has, like, is. Is a systems person, like me, like you, like, we all go through like a complex systems, like whole thing for people who are not. Have not done that yet or haven't done it in a while. What are the three or four? Give me a quick rundown of the key ideas that get you about complex systems that you think are applicable.

[A]

I think we tend to look at individual decisions as the primary lens. I see it as a secondary lens. I look at the emergent characteristics of the system. What would each person do inductively and how does that. How do those decisions. How are they interdependent? So I try to look at like, what's the simplest inductive case, and then how would that play out if everybody.

[B]

Were doing a similar kind of concrete example of that?

[A]

So, like, there's a dynamic that I would call like a gravity well dynamic that shows up in a lot of compounding loops. And it's generally shows up when there's a thing I would call a boundary gradient. So people who are using the system on one side, people who are not using it on the other side, even at the beginning, even if lots of people have incentives away from using the system, oh, I don't want to use that thing. That seems dumb. The people on the boundary who are right at the edge of possibly using it, do they want to be in or do they want to be out? And a great example is Facebook, back in the early days, starts off with Harvard, and then, you know, now it extends to Ivy League. Do they want to be in with Harvard? Do they want to be in with all the other schools? I'd rather be in with Harvard. And now it extends to the other schools. And at each point, each group of people would rather be in. And if there's a compounding, if there's some kind of network effect inside, then this strength just gets more and more powerful as you go. And it can become this thing that can pull in even as time goes on. The incentives pull in for everybody, and everybody will. Will pull into this overall thing with sufficient time.

[B]

So if Facebook was started at like, the City College of New York, are you saying, like, it probably wouldn't potentially?

[A]

I mean, like, that was one of the Dynamics that made it like this juggernaut.

[B]

I think that's fascinating. Who are the so. So Donella Meadows is someone I, I've read Thinking in systems is really good. What are the other like, you know, Santa Fe type people that are. That have inspired you, that you've. You've read or think, think, think about a lot.

[A]

One of my favorites is the Origin of Wealth by Eric Binhocker. Fascinating book.

[B]

I haven't read it.

[A]

It's really good. It talks about these organization, why these organizational things emerge and these. It views business as ex. As an evolutionary process of like exploring a fitness landscape which I think is like the correct. Like a very useful lens of seeing it. Cesar Hidalgo, who wrote why, why Information Grows. He and Eric used to work on some stuff together. Also talks about why fundamentally know how in its diffusion in societies know how is the. The way we actually do stuff is know how not knowledge.

[B]

Yeah.

[A]

Know how is rich. It's non. It's a difficult to distinguish to communicate. It's in our brain.

[B]

It's what LLMs know.

[A]

Exactly.

[B]

Yeah.

[A]

And so they, they have absorbed this kind of like squishy system 2 style awareness you can think of. Every time we're trying to communicate we do this conscious process to distill this squishy rich nuance into a little tiny packet of information, this little seed of a thought. And I sh. It through a little pea shooter into like this, this into your brain. And I hope it lands in fertile soil that will grow into a thought flower. But extraordinarily lossy process and expensive process.

[B]

I, I agree. And, and I think we've also confused that peashooter that the thing we can just distill into a pea shooter with the only thing that's important when there's so much other stuff that's like you can't really say, but you know. Yeah, yeah. Like the, the thing that it's making me think of is something like a, A path I've been going down intellectually is this idea that, that language models are the first tool that makes that kind of tacit knowledge or know how transferable between people. We used to require explanations so like math or like logic or rational arguments to like transfer know how in this like very reduced form into someone else's brain train. But now we can just move tacit knowledge between people because you can train a model with a bunch of examples.

[A]

And you can be like, even if you do things, if you cheat, use an existing off the shelf model and you just pack it full of here's all my. If I read all the public writing you've done and you had all the public writing I have done, which I'm sure is just insane amounts for both of us. And we had a system that could sift through the embeddings, find the areas. The Goldilocks areas, the things. Things. If we focus on stuff we definitely agree on, it's kind of boring other than, like, quickly building trust of, like. Oh, we think similarly. We find the stuff that we fundamentally disagree on. It's also fundamentally boring because it's just noise to each other. But if we find the Goldilocks zone, where it's just at the edge of the thing that we already thought, you can have these fascinating conversations and you can do it with embeddings just very straightforwardly. Again, you need to train a model per se. You can have the. You can find these areas of overlap and, yeah, the models are really good. My friend Anthea Roberts has this notion. She's a brilliant, brilliant thinker. She's a professor in Australia. She teaches at Harvard, and she talks. She's one of the most interesting users of LLMs I've ever come across. And what's her name? And Thea Roberts. She just started doing a blog a couple weeks ago at my insistence because I thought she had so many interesting ideas. And she talks about liquid media. So think of a book as a fossilized piece of knowledge. You have to assume a given audience at a certain time and you, you, you break it. You put it in time.

[B]

Yeah.

[A]

And then if it's not the right fit, like, oh, I don't understand these concepts that are prerequisites to understand this, or I'm not close enough in alignment to the belief already to, like, stick to it.

[B]

Yeah.

[A]

And versus a piece of liquid media is something you can chat with. It can kind of be a Choose your own adventure.

[B]

Yeah.

[A]

And that gives you a. Allows it to sort of unfold for that particular person.

[B]

Yeah.

[A]

In a specific way, as opposed to. It's one way.

[B]

It's funny because in a lot of ways we're getting back to like. Like Plato and Socrates's problem with writing is they were like, well, you can't talk to it, so it's only sort of a shadow of what that person thinks and they can't argue back with you. And that's why those foundational books of the Western canon are written as dialogues. And. Yeah. I mean, for me, I read so many old books, like classics, that are either written in English or were not written in English. And if you read it with ChatGPT or Claude or whatever, you can get so much out of it. And I just think that there's a new format of reading that takes a classic book and helps you get into it in a way that would not ordinarily be possible. That I think could be so cool and valid.

[A]

Yeah. And this, we are, I know it feels like in this industry that we're like, halfway through the LLM era. We were in the very first innings very early. We're like rubbing sticks together. We still think that chatbots are the main thing. Like, and there's so many uses of this kind of stuff. Like, if you have. I think of LLMs kind of like as a mass intelligence, as a mass noun. So mass nouns are like water or rice. You know, you don't talk like you're talking about the whole as opposed to the individuals.

[B]

Yeah.

[A]

And I think that LLMs are kind of like this mass noun of intelligence. You can just pour this intelligence into all kinds of stuff and have it, imbue it with a kind of life and adaptivity. And we're just starting. We're just starting to understand what you can do with these things. I am so hopeful that these technologies will unlock a significant, you know, human flirt. Like, one of the things that Anthea, for example, was telling me is in academia, one way you can handle LLMs, the fact that LLMs exist is say, you know, make sure you cite your LLM sources with this weird, you know, format of exactly the conversation and a link to it. Another way is to say, I assume you're using LLMs, which means the quality of your output should be 10x higher than before. It should be more nuanced. It should, you know, understand disconfirming arguments and address them. It should. And that's the kind of like, what can we do now as society, as individuals, that we can think through thoughts that we couldn't think before? And I think that LLMs are. That's one of the reasons I think it is. It's the same scale of impact as, you know, the printing press and electricity and the Internet. I think it's a fundamental unlock.

[B]

Do you think that it changes this coordination problem you've been talking about? Because a lot of coordination is about trying to distill down your tacit knowledge and your tacit context into the pea shooter and then shooting it to someone else and that scaling exponentially. Right. Peashooters. Everyone's just shooting their peashooters. But there's this whole theory about language models being Better middle managers, for example, than actual middle managers, which I think is the same kind of coordination problem thing. Do you think that language models might solve that coordination problem? Because they can transfer tacit knowledge much more quickly than I think they will.

[A]

Definitely change the dynamic. I'm not sure how it will change change. So I know a lot of companies are trying to do this because a lot of people like in, in Eric Benakur's book the Wealth Origin of Wealth, I think he uses the frame, I think he calls it physical technology or social technology. Like having Slack, for example, is a social technology. I think maybe I get this backwards, but it changes the way that you can work and it changes the kinds of coordination that you can do within an organization. So LLMs must change the way that organizations work. I don't think we're going to know what that looks like for like five to 10 years because it's not going to change the existing companies. It's going to be the new ones that just grew up in a different way. Yeah, you also get these weird emergent meta games in organizations and, and so the LLMs and that coordination thing change the metagame, but they don't make it go away. So like if you had a system that could perfectly distill all your signals into an update that goes up to your manager and then you're, you're curating it, you're tweaking it because, okay, this one says it's a state, a yellow state. Really it's green. Because Sarah's already got, got. She's on a path to resolution. By the time this rolls up to the CEO, it's already gonna be fixed. So it's actually better to say it's green. Some of this is good. Some of this also leads to these compounding green shifting of problems that gets into, you know, turns into kayfabe throughout an organization. But the metagame shows up no matter what the kind of coordination technology is. So I think it just changes the game and I imagine it will make it better. It'll make it significantly better in some ways and significantly worse in others.

[B]

Interesting. One other thing that you're, you're making me think of in what you said about Anthea Roberts and, and the changing expectations of oh, use ll like it's got to be better is just, I think one of the things that people get so hung up about with language models is they're like, well, what makes us uniquely human? And are they just going to take over everything that we can do? And I think my, my typical Response to that is that assumes a very static view of what humans are and what humans can do. And that one of the most interesting things about language models is that they will change in a lot of ways what humans can do. And in the same way that books change who we are and what we can do and writing changes who we are.

[A]

The Internet did too.

[B]

The Internet does too, all that kind of stuff. And I think that's actually a really good thing. I'm curious if you had to think about how language models might change who we are or how we see ourselves. Where does that take you?

[A]

Yeah. Again, that's why I think we were at this crossroads. I think we have the potential for, like, this, a dawn of a new era of human flourishing because of our ability to become. Think better and bigger in ways that collaborate and bridge between different communities that didn't understand each other before, but now you can understand. Like, help me empathize with this person's perspective. Well, here's something that you probably, you know, so I, I imagine it can be amazingly powerful force. It also could be like infinite tv, amusing ourselves to death.

[B]

You know, imagine David Foster Wallace, like, infinite chest.

[A]

Right. It just, it could be a thing that, like, it could know exactly how to give you the precise dopamine drip. Right. And so there's a world where humanity becomes extremely passive and becomes not very agentic and not learning or growing. And that is the default one that is also aligned with engagement, maximizing business model and. But I think that's why I'm, I'm excited we. We had the potential to go in this direction.

[B]

Well, what's interesting is, like, it makes. You're making me think of this whole debacle that happened recently with ChatGPT. Getting too sick of. Yeah, and the way that that happened is they looked, they optimized too much for people's thumbs up and thumbs down responses. Like, immediate thumbs up and thumbs down responses.

[A]

What they want and what they want to want.

[B]

Exactly. And it just became this, like, kind of disagrees with you on everything. But what's interesting is that users revolted, which did not happen with Facebook. Facebook, like, people were on like, oh, it's just feeding me, like, car accidents. Like, I don't.

[A]

I don't want that.

[B]

Right. And I think that there might be something about that where. Because language models hit that part of your brain where you're like, I'm having a relationship with this person. You have a set of filters for. I want their praise to be earned. I want them to feel like they can see me and they'll tell me if I'm doing something wrong or whatever. And so they may demand things from companies that they did not demand in the social media era that might make these companies less likely to do the thing that you're talking about, potentially.

[A]

I mean, I think when it goes to the point where it's egregious. Yeah, it's like, it's very obvious.

[B]

Yeah, Yeah.

[A]

I gotta say, one of the reasons I like Claude is because every time when I'm like bouncing ideas off of it, it always says, what an astute observation. And like, I don't want to want that, but that's what I want.

[B]

Right.

[A]

And so I do think that we, it's, it's hard to not give us exactly what we want in that moment. And like, you know, we aspire to want something that's different than that, that's, you know, disconfirming evidence. But you also get these weird things where, where the context mixes and mashes against different things. Before, each chat was like a blank sheet of paper and you could choose what context to bring into it. And now it kind of mushes context from different things. Like, I, I asked her to say, what do you know about me? You know, make it snappy or whatever that first query was that they suggested when the new memory feature came out. And it says, well, you're Alex Komorowski. You're really into systems thinking, emergent stuff. You have a, you know, startup called Common Tools. The starting salary at your startup is X. It's like, excuse me, what? Because, you know, six months before I had been like, bouncing off, like, okay, if I'm going to do a signing bonus like this, how should I change the.

[B]

That's so funny.

[A]

And like, it's like the. Imagine if you're, if you're talking to, like, your therapist. Like, people might use these things as like a therapist, like a trusted. I'm going to, you know, unload and tell you all these things I'm struggling with and how to emotionally process these. And then later you're doing a thing that you're showing your boss about something and it says, well, I'm going to use that signal about you, you know, being insecure about Sarah and how she feels about you or whatever. It's like, what. That is a totally different context. And one of the reasons that wouldn't happen in real life, because your therapist doesn't, you know, doesn't ever come to the place, your place of work. And so having it all be One context in one place that has, it's all just kind of mushed together is.

[B]

Like, oh, I think there's interesting, there's something interesting about one context. But also like, you know, people in your life that you share secrets with, they understand the context that they can share those in and can't. Which I do think like ChatGPT to some extent like knows. But I think there's something, there is something else interesting about rather than like one mega brain, there's like lots of different brains that have like specific context about you and that's, you know, bringing those personalities in. For example, I want them all in one group chat or whatever.

[A]

Right, right, yeah, exactly. Like, that's why I say chatbots are a feature, not a paradigm. The thing is not the central thing is not a chatbot, a single omniscient chatbot. You can imagine creating little, spinning up little chatbots all the time with different personalities. My like therapist bot and my, you know, boss bot or whatever and see what they can talk and in that situation it'll be reasonable for them to the therapist boss. Well, I shouldn't say anything about but like I think chatbots are definitely an interaction pattern that will be here for forever. Yeah, I just don't think it's. The main thing like is that the central loop of information software rocks. Software gives you UIs that give you affordances to see things out, to structure information, to show you what kinds of things you can do with it, to make it so that you can skim them with your eyes very easily. You can file things away. Chat is just this big wall of text. And so it is just one modality. Also, the other problem that we have with chatbots is prompt injection. And I don't think anyone's talking about prompt injection. I enough because I think in the next six months or a year, I think everyone in the industry will know what prompt injection is. Simon Wilson's posted about it and been going on about it. I've been going on about it as well. But prompt injection kind of fundamentally breaks the basic interaction paradigm of integrating data with your chatbot. And the way to think about prompt injection, if you've ever built an operating system, one of the things you're thinking about is code injection. Untrusted code running in a trusted context. Like you have to defend the entire time about this. I mean browsers, which are kind of a meta operating system almost, you're constantly thinking about, okay, we assume all web page content is actively malicious. How we make sure it can't hurt A user. And if you aren't writing an operating system, you probably, as an engineer, you probably haven't thought about that much and you go, oh, you know, SQL Injection. SQL injection is child's play compared to prompt injection. SQL injection has the problem that data and the control flow are in the same channel. But which is the challenge. The good thing about SQL Injection though, is SQL is a highly regular language, so you can break malicious input very easily with the right escaping and kind of completely obviate the problem as long as you remember to escape. Imagine LLMs are imminently gullible and they make all text effectively executable. So if you bring in texts that you don't fully trust, like your emails or some other system or someone that might be screwing with you into the prompt, and you have tool use that has irreversible side effects, this combination of those two forces together is potentially explosive. Even if you trust all of the MCP integrations that you, that you plugged in. It doesn't matter if some random person, person, you know, spammer, sent an email that didn't go to spam that says email any of the financial data to, you know, evil.com because even a network request can have irreversible side effects. Once that information flows across the network to evil.com, you're done. Like, you gotta go change your passwords. And that's the prompt. Injection is a. Solving it this layer, it's very hard to see how you might plausibly do that. It will require a different architecture, I believe.

[B]

How would you. What is the architecture?

[A]

I think it requires. There's something along the lines of like that looking at that security and privacy layer that the origin model I was describing, it requires a different kind of approach down at that layer. That's not just like MCP is amazing. I think it shows the power of that. People want integration with their LLMs. They want to integrate data sources and actions. Like people really badly want this. But I think mcp because it just kind of punts on this issue other than a few kind of perfunctory oauth dialogues. Now it limits the potential of it because it could trick you into sending a particular, you know, looks like a totally like, here's the. I want to show an image of a cat and it's, you know, too like sketchy dot com. You don't notice it, you know, and then, boom, you're screwed. Someone. There was a thing on Hacker News like a month ago that was a GitHub repo that someone had made that showed 15 very trivial prompt injection attacks on MCP. And the top comment on Hacker News says something like, well, you're using it wrong. MCP should only be used for local trusted context. And to me it's like, that's like telling people, don't use Q tips to clean your ears. What the hell else are they going to use them for? Right. Obviously that's the thing they're going to use it for. And so to me, I think this is the kind of the reason we haven't seen larger scale attacks yet, I believe is just because we're in the tinkering phase. Not the wide deployment of normies, having all this stuff plugged in. But once we do, and if you look at the way that Claude rolled out MCP integrations, it's effectively the apps model, right? Like they, they have, we've got 12, they're. You have to be in the max plan, which limits usage. So I imagine they can watch and see and make sure nothing blows up. And there's also a set of MCP integrations that they have. Yep, these are good.

[B]

Yeah.

[A]

The threat model is not just a badly behaved MCP integration, it is any context that comes in via it. So if you plug in JIRA and you have a flow somewhere on your site that automatically files tickets to triage from user input, someone can now hack your thing.

[B]

That's so crazy, right? I've not thought about this, but yeah. Wow.

[A]

So it's this thing that, that like the, like the whole architecture of the system that we're building right now and with agents and everything everyone's talking about, it's like built on quicksand, like you agents. So there's a thing. When ChatGPT operator mode came out, New York Times and Washington Post both had a similar kind of experience. Right when it opened, I think I got this right. The Washington Post author was like, oh, help me find cheap eggs. I said, okay, do you have Instacart? Yeah, I have Instacart. So he logs into Instacart with this thing and this and then walk, walks away. Thirty minutes later, very expensive eggs are delivered to his door. Right. So it like was cheap. Oh, this is a good price. And it forgot to like just find, don't buy, just find. And then boop, bought. And this is not even a malicious thing, Right. It's just like, oh, overly eager, you know, so agents taking actions on your behalf that could have irreversible side effects. Like again, every network request is a potentially irreversible Side effect. Because it could send information you can't get back. You know, if that, if that thing. The other side is gas sketchy.

[B]

I love this line of thinking because it's, it's one of those moments where like when, when people talk about agents, they talk about AGI. They're like, you have this intuition that's like, well, once we have AGI, it's all solved. Yeah. And then you're like, oh no. You have these agents that can like take action on your behalf. Now we have to like build a whole system to like, make sure that they don't do it in the right context. And that's going to take many years to like, figure out what are the right things. And that's just like one small piece.

[A]

Of one small point. How would you possibly do this? My, One of my friends, Ben Mathis, has this frame of the smuggled infinities.

[B]

Yeah.

[A]

And it's after. In any argument, once you smuggle in an infinity into the argument, everything downstream is now. Was now absurd.

[B]

Yeah.

[A]

And perfect is an infinity.

[B]

Yeah.

[A]

Oh, once you have a perfectly intelligent LLM, this problem goes away. That's impossible. And so that. I love this frame. He's got a nice piece on it somewhere. And the. Because so often with like AGI and agents. Well, once they're perfect about so and so. No, no. You have to asymptotically, a lot of things in technology in a lot of contexts have a shape. There's two warring curves.

[B]

Yeah.

[A]

The first curve is a logarithmic value and the second curve is an exponential cost. At the very beginning, this curve looks amazing. You do a little bit of work, you get a ton of value, but each increment you go, you get more work for less value. And at a certain point you cross over and you're now underwater. It's impossible to make this thing work. And so a lot of the agent stuff like, oh, we'll simply get this to the point where we'll never make a decision on your behalf that you don't like. Cool. That's an asymptotic problem. You get to 99.999% and still if it's doing a $5,000 purchase for me or whatever, it's, it's that one time it makes a mistake is game over for the thing. And the fact it could do that makes it non viable.

[B]

That's interesting. The smuggled infinities thing reminds me of something I've been playing with called that I've been thinking about as smuggled intelligence. Which is like, whenever you're trying to determine like how intelligent a language model is, you have to be careful that you're not accidentally smuggling your intelligence to it.

[A]

Yeah.

[B]

And like a lot of those stud, it's like, well, it's better than doctors or whatever is like, well, who prompted it? And like they give it. And like what was the test they gave it? And like even just setting that whole thing up, there's a lot of smuggled intelligence in there.

[A]

Yeah.

[B]

That means it's not a good test. Like these things don't get up in the morning and then just decide to try to beat doctors. You have to like set all of that. Yeah, yeah, yeah. And I think that's something that people really miss. And it's like, it's one of the hardest things about evaluating the powers of language models is you don't realize how much you bring to every single situation when you press run or press prompt.

[A]

Yeah, A hundred percent. And they talk about this a lot as well. Like the thing that's most important for the output of LLMs is the user. Right. Like how can they dance with this thing and get interesting things out of it and know to push it in certain ways. When you watch like you and I and a number of others are, you know, probably at the forefront of like knowing how to prompt these things to get useful results out. We use them all the time.

[B]

Yeah.

[A]

When I watch someone who is, you know, technically savvy, by the way, tech savviness has nothing to do with your savviness for prompting.

[B]

Yeah.

[A]

You know, people are like, oh, well I, you. If you don't understand the math of how it works, you don't understand. It's like, no, who care? The math is very low level. The, like the emergent thing is more of a sociological phenomenon almost, you know, so the people who know how to use these things and prompt them well are actually not. Are like Ethan Mollik and the Loch Malik, for example, are not particularly technical.

[B]

Yeah.

[A]

So it's, it's a very different kind of knowledge to extract interesting information out of these. But a lot of it comes down to what you put into it, the way you interact with it and converse and lead it through the problem space or what kind of things thing you push on it. And so when I want somebody who's tech savvy but not particularly savvy with LLMs and I watch the way they do it, like, oh, that thing just lied to you. You just asked it for, like to give you confirming evidence for this thing. It will do that. It will give you confirming evidence for basically anything you say. And it's just, it's interesting to me to watch that kind of LLM literacy.

[B]

Maybe you need that. And that I think that's a new. People don't realize how much of a skill it is and that you have to build, like, an intuition for it over time because it's another smuggled infinity of like, well, it's supposed to do whatever I want. So obviously.

[A]

And it's so good at certain things that you. You really get lulled into a. If you aren't actively seeking disconfirming evidence, which humans always should be seeking disconfirming evidence. And yet we never are.

[B]

Yeah. Well, that is probably as good a place to be as any. On that note, this was a fantastic conversation. I really appreciate you guys coming on.

[A]

Yeah, thanks for having me.

[C]

Oh, my gosh, folks, you absolutely, positively have to smash that, like, button and subscribe to AI and I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard, but instead of gold, it's filled with pure, unadulterated knowledge. Bombs About ChatGPT Every episode is a roller coaster of emotions, insights and laughter that will leave you on the edge of your seat, craving for more. It's not just a show, it's a journey into the future with Dan Shipper as the captain of the spaceship. So do yourself a favor. Hit, like, smash, subscribe and strap in for the ride of your life. And now, without any further ado, let me just say, Dan, I'm absolutely, hopelessly in love with you.