← The Tech Policy Press Podcast
The Tech Policy Press Podcast
Why the AI Policy Debate Should Focus More on the Harness and Protocol Layers
00:47:21
Loading summary
Transcript47 lines
- [00:01]
A
Your film is now ready to be shown.
- [00:12]
B
Good morning. I'm Justin Hendricks, editor of Tech Policy Press. We publish news, analysis and perspectives on issues at the intersection of tech and democracy. My guest today is Rathi Krikorian. The Chief Technology officer of Mozilla Rafi has spent the past few months building an argument I want to dig into that a central question in AI isn't open versus closed, but owning versus renting. Whether AI becomes something we can control or something we lease from a handful of companies. He's a technologist by background with stops at Twitter, Uber and the Democratic National Committee.
- [00:46]
C
And he writes about all of this
- [00:47]
B
on his own newsletter owners, not renters. And in other outlets, most recently in a New York Times op ed on what he calls the Mythos moment. We covered a lot of ground in this discussion. The idea that generosity is the hidden infrastructure of the Internet. How to expand access to powerful AI tools rather than closing them down for security's sake, how to overcome misaligned incentives to build a better information environment, and how to counter surveillance. Let's jump right in.
- [01:12]
A
Rafi Krikorian, I am the CTO of Mozilla Rafi.
- [01:17]
C
I'm excited to talk to you. I've had the opportunity, of course, to meet you in the past and your role at Emerson Collective, where I had the chance to be a fellow a few years ago and appreciate the fact that you invited me on your podcast a while back. So I'm so pleased to get the chance to return the favor. I'll also say, you know, this podcast, my listeners know we host a lot of tech criticism. And while I know we'll have a critical conversation today, you also know are, I think, fundamentally an optimist. You know, that is, I believe, what your podcast was literally, you know, falls,
- [01:58]
A
if I'm correct, technically optimistic.
- [02:00]
C
Yeah. And I think it's good also maybe to do a level set on where we're at on some of these technologies, particularly with artificial intelligence, and what policy mat makers should be thinking about when it comes to these things. I sort of want to start with your New York Times op ed recently. You sort of laid out a sort of open source thesis, this, you know, kind of vision of the implications of the Mythos moment, what it means for the broad enjoyment of technology's benefits that we've experienced, or at least some would frame as what we've experienced with the Internet over the last few decades. Why was the Mythos moment important to you? Why did it strike you as a moment to pick up the pen yeah,
- [02:47]
A
that's a great question. So, like, let me give you some background of where I was coming from. So Anthropic released Mythos, or released it to a small set of companies and organizations. Fortunately, the Firefox team at Mozilla was one of those organizations. And what Mythos did for us at least, is it found a slew of exploits. And, you know, the Firefox code base is pretty old. Not old in the sense that, like, it's janky and crusty. It's old in the sense that it's been an old open source code base. There's been. I can't even count how many contributors have looked at every line in this code base. And Mythos comes around and discovers all these vulnerabilities that humans have just missed in the decades that that Firefox code has been around. And so that, for me, was this aha moment of, like, oh, no, this is different. Something just happened where this balance in open source has changed. So in the sense that, like, software up until this point, or up until, like, what's called November of 2025 last year, was hard to write. And right now, we're living in this, like, this golden age of writing software, where software writing has become free all of a sudden. Like, you know, a dentist can describe a piece of code they want for their website, and, you know, some agent will just write it for them. But now all of a sudden, what Mythos showed us is possible is that finding bugs has also become free. And so, like, the world has existed in this, like, this balancing time where writing code was hard, but finding bugs was hard. So, like, we lived in this weird detente that, like, not many people wrote code, but not many people found bugs in the code that's being written. And, like, the entire Internet is now hoisted on the shoulders of open source code. Like, there's code like FFMPEG that, like, is in every single web browser, but it's probably also in YouTube and Vimeo and all these other streaming services. It processes video, and it's like the de facto tool you would use to process video. Or, like, curl, which, like, fetches web pages from the Internet. Like, my son's PlayStation has curl on it. Like, every single desktop computer probably has some variant of curl in it. And so all these pieces of code that authors have painstakingly written over the years, which now buttress the Internet, Mythos is coming around and finding vulnerabilities in all of them. And so the fact that this equation has changed. Software is now easy to write and Bugs are easy to find. Could be a pretty bad thing, or it could be amazing. It could be a pretty bad thing because it means that attackers could come around and find bugs in all this critical software that holds up the Internet. So for me, the point of the piece was to a point out that this imbalance or this balance has just become an imbalance, but also the social contract that we use to hold up the Internet, that a few people through generosity have written large amounts of open source software that now power the web. But though that generosity is under attack, like, there's only two people that maintained open ssl. And a few years ago we saw a bug in open SSL called Heartbleed that caused us all to, to, to question the security of the Internet. But it was two people. And so, like all these companies are spending all this time and energy writing software, but what's really holding up the Internet is generosity. And Mythos for me, exploded that generosity.
- [06:20]
C
So the implications of that are it's a closing up, a kind of, you know, security moment, a securitization moment on some level. We have a piece from Konstantino's Kamitis this week who's talking about AI as potentially sort of, you know, introducing a new logic to the Internet as well, kind of a similar line of thinking, I think, on some level, at least some degree of congruence. And he's sort of imagining this world where cybersecurity eats up all the questions. You know, everything becomes about cybersecurity, everything becomes about defense. Is, is that sort of similar to what you're describing as the worst case scenario here?
- [06:59]
A
Yeah, potentially my worst case scenario. But again, I'm an optimist. Let's think about it from a different angle for a second. So we have to ask ourselves, what's the steady state that we want? And then what do we do in this transition state to get the steady state? So I think one of the worlds I want is that all our software is just secure. We're actually building software that's secure by design. Caesar did this whole push in the last administration for software that's secure by design, so I would love to get there. That's minorly in conflict with all these, with my dentist vibe coding all of a sudden. But we can probably get there if we do a concerted effort. And then I want to get to a place that our intranet is secure, that we've actually found all the vulnerabilities and we're living in this way, better spot than we're living at today. That we're living this slightly constant fear every single day around this critical infrastructure that maybe we get to a place that that's all secure already because Mythos and Mythos, like tools have found all those bugs that would be an amazing world to live in. So the question is, what do we do right now? And so, like, a lot of people are making this argument that for security stake, we need to actually close up access to all these models because they're dual use. They can be offensive and defensive. And what I would argue instead is like, we need to do a concerted effort and investment that we should be like, you know, Anthropic pledged something on the order of $4 million to open source communities to use advanced tools like Mythos to help define bugs. Imagine a world where every single company that relied on open source did some form of contribution to accelerate the fixing of all these bugs and the deployment of all these bugs. Like, I want to get to a world where it's not security isn't causing us to shut down Access, but in fact we're. We can use this moment to accelerate access to the right people under the right measures, so we can actually get to a world that's secure. I think that's the world I want to get to. So I'd rather think about that than the worst case scenario.
- [08:57]
C
One of the other things that's been discussed on tech policy press around this Mythos moment is the kind of general sort of inequality. You know, who's been given access, who's in the know about what the reality of the threat actually is, who gets to sit around the table to talk about what should be done. I don't know. Is that part of the issue here too? I mean, you're. You're one of the people who, I suppose, maybe not you yourself, but your teams have, you know, had the opportunity to at least get under the hood here.
- [09:26]
A
I mean, I agree, like, you're onto something there in the sense that, like, it sort of like matches my line of, like, we need to expand access, not restrict access. Like, security in my mind shouldn't be something for the verified. You like? I'm very grateful that Anthropic partnered with us to find all these bugs in Firefox. I think of Firefox as an important, almost digital public good in the world that needs to exist, needs to be secure. People can trust things like Firefox or Firefox specifically. But at the same time, I think you're right that, like, we can't be restricting it only to the rarefied few. The security tax that sort of exists today is generally a bad thing, right? Like if you go to any piece of enterprise SaaS software, if you want to get SSO single sign on, you want to get two factor authentication, you want to get all these things like these companies are asking me to pay more money to get access to features that will make me more secure and I think that's a bad anti pattern. Like we need to figure out how that security is by default given to all of us in some way, shape or form. And so like access to Mythos is one of those access to Mythos and Mythos like tools like, you know, there are a bunch of open source projects and GPT OpenAI now has their GPT cyber. So there are a bunch of different initiatives to try to build tools to do this type of scanning. We need to make sure that like the right people actually have access to them across the board, whether it's even just one person who works on Curl. But again that software is on every single PlayStation. Like we need to make sure access is more uniformly stripped. It.
- [11:04]
C
I want to also just ask you about this idea that you have brought up in past around this problem of kind of like renting versus owning or the kind of rentier economics of AI that we appear to be, you know, building into the ecosystem. You know, this sort of feels like it sort of stretches from that a little bit that, you know, I want to get hooked on a handful of providers who are going to give me access to the main line, the best possible, you know, security agents, etc. Etc. And you know, I'll just have to wait until they upgrade their product. I can never really own my own security. That feels like a problem.
- [11:44]
A
Yeah, I mean the renters versus owner's frame I think applies here too. So just to make it just to stay what it is, I generally believe that, you know, in the last, let's call it 10, 15, 20 years of the Internet, we've slewed toward this model of renting our access instead of owning our access. So like we see this right now in things like the way that we access these intelligent services, right? So OpenAI's ChatGPT, Anthropics, Claude, et cetera, we are renting access to them, right? Like we pay n dollars a month to get access to these models who honestly, their incentives are not necessarily aligned with mine, they're aligned with making more money. We see in the way that these, these models are talking to us. They're designed to actually get us to engage more We've seen some horrible things about the way that chatgpt and I'm not implicating them, just sort of stating the horrible things that like ChatGPT has said to children, that is, that shows that their incentives aren't aligned. There was a paper that just came out between Princeton and UW which has like solidified what we all were thinking, that if you ask one of these agents for a recommendation on some good to purchase that they're more likely to recommend a sponsored thing versus what you might actually want or they might be doing some form of like social economic modeling of you in order to try to see how much money they can extract as opposed to exam giving you the thing that they want. So we're seeing all these systems which are designed for the profit and motivations of the providers versus ourselves. And again, this is not new, right? We saw this in the social media era. We saw this before Google was putting sponsored as a note on their on their tags when they do PageRank. So like this is a pattern that keeps on showing up. We're just seeing the latest iteration of it. And so when I say owners, not renters, I want to get to a world, especially if we're going to get to a world of AGI, which I know is a very American centric way of looking at these open source technologies. But I don't want 7 AGIs, I want 7 billion AGI. It's like I want an AGI that's truly on my side and taking care of my interests, not in someone that's making decisions on my behalf, like someone that I can share confidential information with, I can share healthcare information with, I can share all these things with that. I know it's not being used as a surveillance mechanism. I know it's not being used to profit someone else's bottom line. And so like that's what I mean by this owner's not renters frame. And so bringing it back to security. I think the same thing is happening here too of like I'm not getting stuff that's architecturally secure. I'm getting security through the fact that I'm paying another vendor. So a good example here is to think about the way Signal operates, right? You know what Meredith and her team Moxie, et cetera have all built is something that even they can't snoop in on the messages, right? Like if a police department of FBI, if a foreign national comes by and asks Signal for the contents of Rafi and Justin Signal transcripts, they literally cannot give it to them because it's just architected in a way that's secure. And I think that's, in my mind, how ownership should work is that we're getting things that are architecturally safe for us to go use. And we are so far away from that in this AI world. And that's the world I think we all need to be chasing more.
- [15:14]
C
Well, I do want to kind of ask you a few questions about sort of what you see ahead for certain technologies with artificial intelligence. But I think many of my listeners, again, you know, who are accustomed to critical AI discussion on our podcast, they hear you talking about AGI, they hear you talking about someone that probably immediately pegs for them. And in their, in their minds, you know that you are, you are definitely on one end of the spectrum in terms of where you think the technology might go. I mean, do you think that's where we're headed, to some form of artificial general intelligence?
- [15:48]
A
Well, I mean, like, the definition of AGI is such a hazy one that, like, if I were being super precise, I put my engineering hat on. Like, I can't say for sure that's what we're aiming towards or that's where we're headed. But what I can say is that if you look at consumer data right now, people are generally switching to these types of interfaces. Like, people want an interface to information that seems to be more conversational or seems to be more able to take in a lot of context, or seems to be able to be answering questions or, like, help you automate workflows. Like, I was just looking at Google search query traffic over the last 2412 to 24 months and like, it's not going down, but definitely took a dent. And if we look at the number of people that end up on a Google search page and don't click on anything else, partially due to AI summaries on that page or maybe something else like that seems to be on the rise. So people seem to be craving these other types of experiences. So if I were to speak for myself specifically, like, you know, I'm running the Hermes agent right now on one of the computers in my rack. Hermes is written by a company called Noose Research, is an open source agentic framework. And it's kind of great. Like, I have to admit it, like, you know, I have it wired up in a very specific way to make sure that it can't do anything super crazy. Like, it's not going to send an email on my behalf, it doesn't have access to my bank account. Like, the people who are Doing that with openclaw are crazy, in my opinion. But, you know, I can text my MREs agent a picture of what I just ate for lunch and it knows what's my fitness tracker that I use and logs my food for me. Right? Like, I can ask it questions of just like, when do you think I can go visit the doctor? Like, if you look at my calendar, the next few weeks and it gives me all my open slots. So, like, I do think that people are craving that kind of way of. Of integrating information into their lifestyle. But, like, is it. Is it something I'm gonna call a someone, or I'm gonna call it a something, or is it something that's truly AGI? I don't know. I have been, maybe as a last footnote, I have been really excited and enamored by the scientific breakthrough work that's been going through. Like, the people who have been pointing some of these really large models around genome science, et cetera. That's been super interesting to me. Of course, open questions around, like, are we actually generating more science or are we just generating more things to be tested in web labs and stuff like that. But there is something here that we all need to be paying attention to more carefully. And I think there's an opportunity, like I did with my agent, that we have to have questions around permissions and boundaries and how it actually interfaces with the real world. So we need to still sort through.
- [18:40]
C
So I have so many different questions that stem from these issues around agents in particular. And, you know, you make me think that I once heard Kate Starburg at the University of Washington, you know, say that, you know, her field, human computer interaction. You know, if you look kind of look back at the beginnings of it, the whole idea was we want to talk to computers. That was. That was kind of like fundamentally where everybody wanted to get to, you know, most of the time in the middle. From that point to today, we've had this priesthood of computer scientists, developers, coders who knew how to talk to computers. And effectively, the priesthood is broken. That's what's happened perhaps Since November of 2025, as you try to hazily mark the date there. But I guess I want to ask you, do you think that something has fundamentally changed with quote, unquote, agentic AI? Is this just software by another name? Is this hype, or do you think there's something legitimately new here?
- [19:40]
A
I mean, you're not going to ask. Answer my. Sorry, you're not going to like my answer, because it's yes to all of the above in some way. I mean, the biggest difference in my opinion is that we are working with these, like, probabilistic systems. Like, you know, the engineer in me still gets like, shivers and chills. I'm being like, I don't exactly know what's about to happen when I type it and ask it to do this thing, which is kind of crazy, but in some way then maybe that's the way the world works. Like, we're all searching for metaphors on how this all plugs together. Like, if I were to ask another human to do something, like, I don't exactly know what it's going to do. Sometimes it surprises me, sometimes it disappoints me. We have to wonder about what that contract is supposed to look like and what that really looks like. One of the difference here though, is that, like, the fact that we're allowing these systems to get access to way more information than we would have traditionally allowed software to get access to, and then allow it to use that as part of its context, right? As way to synthesize what that decision it should do is. That is relatively new. Like, I think that, like no one would have predicted or, you know, except for a few sci fi writers, that being able to assimilate all this information, not just generically what's on the web, but all the context of a particular human, and then try to make a probabilistic guess of what to do based on it. I'm not sure anyone would have bet on that. And because we haven't bet on that, we have a whole set of questions that, that stem from it. Like, you know, what's the legal permissions that we want to give this thing that might be acting on my behalf? Like, is it a fiduciary? Is it not a fiduciary? Is it acting with my consent? Is it not acting with my consent? Like, what do we do when things make a mistake? Are there architectural questions we should do to have a permissions boundary around it? So yes, it is, quote, unquote, just software, but maybe with a probabilistic spin on it. It's different because we're now giving it access to so much more context that it can try to synthesize and, and try to make a guess about what my intent looks like in a way that we were just never able to do before. And then finally there is a real big difference, which is just the economics around this are so fundamentally different. The fact that like a single query for me is spinning up a data center somewhere that's a big deal difference that we really need to really think about. And like if you asked me last year I would have said something like the per token cost is going to go down, it has gone down, but the number of tokens has exploded in the process. And so there is still these open questions of economics and architecture that need to be sorted out here that I feel like again we're in this open transaction moment that given a year, given five years will be settled. The question is just like, can we handle the chaos that we're about to cause in these next five years? I don't have a good answer.
- [22:40]
C
And we probably don't know exactly what the economics look like until effectively kind of the market settles out a bit. Like there's a lot of hype and too much money from all kinds of different places coursing through the system that are probably perverting all the economics.
- [22:55]
A
Well, I mean I'll just give you a sense of what it looks like for me and like I'm, I mean I will admit I'm a semi alpha user right now. I do it partially because it's fun, partially because it's research, partially so I can have an intelligent conversation with you, Justin. But like running my Hermes agent do automating a bunch of stuff that I'm doing, like looking at my calendar, like, like all that kind of stuff, like it's like fifteen hundred dollars a month, right? And like that's like that's clearly is not a sustainable price point. So. And like as the market shakes out that might even get more expensive because VC dollars that dried up or we might need to see architecture shift to like running on my GPU inside of my laptop instead of on a remote data center, which might get it down to more like $50 a month, which is still too expensive for most users or it sort of like opens up the question of just like how is the Internet funded these days? So like I think that economic shakeout of VCs drying up, token costs exploding, architecture is becoming cheaper, moving more to local like it's a sort of a thumb in the air how that's all going to play out. But there's so many variables here, that's part of the equation.
- [24:01]
C
So I want to ask you about another, another issue that perhaps Tech Policy Press readers are paying some attention to. But I feel like most folks in the policy community still aren't really sure what you're talking about when you say the letters mcp, but you clearly think that, that something important could be going on here. That at this particular layer, this could change the information ecosystem quite profoundly.
- [24:26]
A
Yeah. So maybe an analogy that I try with people, you can tell me if it works, is sort of like the web right now is powered off of what's called HTTP, the hypertext transport protocol. It's what ships HTML around. So if I bring up a web browser and type in a domain name, like Technology Press, like, I go to that server and it ships me back some HTML, so some content with some structured markup around it that tells me how to render it on my web browser. But like, agents or like these autonomous systems or LLMs in general don't care about the way that the world is structured. Like, they honestly, they just want the information. And so the question is, like, how do we get the information to them in the most reasonable and most efficient way? So people have come up with, or anthropic came up with specifically, but they've donated the protocol, quote unquote, to mcp, the model Context protocol. And what it's meant to do is it allows an LLM to call to get more context. And that calling more to get more context could be it just wants to read more information, or it could mean that it's asking something to actuate something in the world to therefore get more information. So two examples of this. Like, you know, I've had conversations with publications about whether or not they need to not only have an HTML version of their site, but an MCP version of their site so they can better control the what kind of content an LLM might extract from them. And they can set permissions or pricing models. Like, what would an MCP version of Tech Policy Press look like? And like, could you do billing on it? Could you, like, what are all the things that we think about that you can maybe put on that? So, like, MCP I think, is really interesting in the world of, like, can we get access to different kinds of information? Like, imagine a world where like, I had a subscription to the New York Times and they exposed an MCP for subscribers. So, like, all of a sudden my chat agent could use my subscription to get access to more information than what they could get on the New York Times website without being logged in. And the New York Times currently hates LLMs, so you can't get anything but, like, think about that as a, as a particular example. But another way to think about MCP is also tool calling. So an LLM right now can just on its own synthesize a bunch of text. So you give it a bunch of text to figures out what to do with it. But if you allow to call a tool, it could actuate something and then get the response back. So you could ask it to call a tool to like change a design on Figma, and then all of a sudden a user would see a different design in front of them and then might react to it and give them more information. Or there's a pretty entertaining, but I think important website out there called Human AI, which is an MCP server to humans. So an LLM could call Human AI to ask a person to actually do, do something in the world to all of a sudden the LLM is like puppeting humans around in the world. But those humans could be doing things. They could pick up a package, they could hold up a sign, they could go to a protest. So, like, all this stuff could start happening through MCP that is both interesting and terrifying. Like, we need to wrap our heads around a bit better. It's similar to how in the 90s we all went around talking about both the powers and perils of the Internet because all of a sudden humans, the world had gotten flat and humans could see the entire world through the web. MCP is the same thing for agents and the MCP is the same thing for LLMs of like, the world is becoming flat and people are orc. Are enabling them to get access to more information and to actuate change in the world. So we have to, we have to figure out what to do here because this is actually going to be a big fundamental shift.
- [28:15]
C
So this raises so many different questions for me. But I mean, again, this is the Tech Policy Press podcast, so we're interested in questions around policy, questions around governance. So I'm immediately wondering about, you know, permission models, you know, who can audit this protocol, who can see into it, who, who knows what's going on, who has power, who doesn't, you know, who controls the layer.
- [28:40]
A
Thankfully, you're not alone in these questions, right? Like, you know, I've had conversations with people like networking providers, like, people who make switches that go into data center racks. And like, they've spent. And they've spent a lot of money and made a lot of money building traceability into the web so you can understand who's clicking on what, where are they coming from, what they did next. And they're all asking the exact same question now, what do we do with agents, LLMs, MCP? Like, how do we build traceability like that into our fundamental infrastructure? But you're right on the policy side, like, the Human AI thing is like, really interesting. For me, because it is an example of now humans are becoming puppets to computers, which sounds very matrixy, but at the same time it's just like, what if an LLM asks someone to do something that's kind of inappropriate? Like, who's now responsible in that situation? How does that actually work? Like, who are we logging it appropriately so we can do investigation on it later? Like, what is, what is the trust and safety angle on all that? Like, all these questions are both terrifying and greenfield to be answered.
- [29:50]
C
So I kind of want to ask the question about where this leaves us with kind of AI critique on some level. When I think about a lot of the critiques that people are making about artificial intelligence right now, including many that have been made on this podcast, which, for the record, you know, I am sympathetic and partial to many of the arguments that folks make around labor, copyright bias, climate, you know, all of those types of things on some of those are all aimed at the model layer. I don't know if that makes sense. It does, but you're sort of saying, you know, there's, there's this other thing exploding at the protocol layer and it feels like maybe critics of AI aren't really focused on that.
- [30:32]
A
No, that's a really good flag, Justin. Like, I think that, you know, and I can point you to some stats. Like right now, the question is not just the model, it's the harness. It's that agentic framework that goes around the model itself. You know, an agentic framework is nothing more in some ways than a loop around the model. It's like something that asks the model a question, then decides to act on it and ask the model another question, or might take your prompt to, to the harness and ask the model to decompose it and then ask what to do about every single one of those decompose decompositions. There's a lot of evidence right now that if you look at the capabilities of a model, I'll speak about specifically on coding, and we can generalize from there that, like, the models are really good at coding. And every single time we get a new model release from one of the frontier labs, they get slightly better at coding. But things like cursor or open code or terminus, like all these other, these frameworks that an actual engineer might interact with the model on, it turns out when you take that model with that harness and do the exact same eval of like, how well did it do to write some code? Just adding the age, just adding that agentic framework, that harness around the exact same model increases the ability for that model to do about 10 to 20% better on coding tasks. So, like, there's a lot of gain right now in just making these really good harnesses. So the Frontier Labs a kind of don't want you to know about this because they're spending a lot of money in model development and writing the framework around it can be done by smaller companies with less capital. But there's a lot of leverage that's happening right now in that layer. And so you're right that most policy folks, most governments, most of those people who are concerned are all focused on that on that model side. But what we should be thinking about is the harness is what's actually going to do a lot of the governance, it's going to do a lot of the iteration, it's going to do a lot of the tool calling. So when we talk about my job is being replaced, it's because the harness is actually the one that's orchestrating the model behind the scene. But the harness is going to do a lot of that heavy lifting. It's the one that's going to know exactly how to file a legal brief if you're worried about a lawyer, or it's going to be a one that's going to know exactly how to make a calendar appointment if you're worried about as a secretary, about your job job. So, like, the. The model might have the intelligence, but the harness is what's going to do a lot of that actuation. So it's not. I'm not saying ignore the model and look at only the harness, but now it's this combination of the two where the battle needs to be apt.
- [33:17]
C
It strikes me the whole anthropic Pentagon, you know, fight was effectively a harness fight.
- [33:23]
A
You can make an argument that Mythos is just a harness. I don't. I don't think it is. I think that would be an unfair characterization. But if you dig deeply into the Mythos release nodes, part of what they're doing is they're just allowing the model to run for a very long time. So that's a harness question of just like we let the model run and just let it execute, and when it finished, if we have a question, just issue another query and just let it keep on going. So, like a lot of Mythos gains maybe is just because of the way they're running it, not just the model itself. So that's a harness question. So I think you're right.
- [33:58]
C
What else are policy folks missing right now? Folks who want to govern these technologies as I believe you do as well, to some extent aggressively towards avoiding societal harms. What should they be thinking about?
- [34:12]
A
I think we need to do a more examination of what's the system we want to look like in the future and then ask the questions around that. So if we look at it through that lens, let's look at the future and backport it a bit. I want to live in a world where we have open systems and open source. So we have both open source open weight models, which I understand has a lot of trade offs around them. Like whenever I mention this to a lawyer they immediately come back with csam. But I'll also then counter with the benefits of open source and transparency and stuff like that. So clearly a pro and con and a gray zone we need to work through. But I want to live in a world of open system. So if you think about like that agentic harness question, it's not that I don't want the anthropics or OpenAI's to exist, I want them to exist at that extreme frontier of the things that takes a lot of capital, a lot of resources and are potentially the ones doing scientific breakthroughs, et cetera. But I want for commodity usage to go toward more of these open systems that are locally deployed, that don't require a network connection that I can use even on United WI fi kind of thing. So if we think about that world of this heterogeneous mix of closed source, open source all being orchestrated and interacted with, there are a lot of policy questions around that that we need to solve and figure out. I don't even know how procurement works in that world. I don't know how responsibility works in that world. If I'm running a mixture of things in the back end, we need to disentangle which one actually caused a particular action to occur at any given moment. Like I think all these type of questions need to that become exponentially harder in this like mixture world that I'm sort of proceeding. And so like I don't think of any one particular thing we need to look at, but instead this like crazy combination that we're about to race towards. And I don't think, just to be clear, I don't think I'm alone in thinking we're going to get into that crazy combination of the world. Like that world already exists and not AI, right? Like if you look at the way our computers have set up, like Linux kind of runs the world and then there are a bunch of iPhones. But so like that causes a bunch of complexity in the world, there's heterogeneity, intragenality in just the way the web is served at any given moment. So I think it would be, like, unfair not to say that we're going to get to the same world in AI. So thinking about that complicated system is going to be the crux of it. And if you think about it that way, then the harness layer becomes super important, tool column becomes super important. That interaction of, like, models puppeting people becomes super important. Like, those type of questions take a
- [36:48]
C
little bit of a detour just to ask you about, you know, some of your views on, again, the world we're building, but maybe from a different perspective. One thing you wrote, or I guess a question you posed on your blog, technically optimistic a while ago, stuck with me. You wrote, if you're being recorded every time you go outside, are you living in public or are you living under surveillance? And if there are any close listeners of this podcast, they know that I brought this up a few times about how in my neighborhood in Brooklyn, enough people have those ring cameras that whistle at you when you walk past and tell you you are being recorded. So you, you let me, let me see if I can do it. It's something like, you are being recorded is the little message you hear. And I must hear that when I'm walking my dog, you know, every three or four houses, right? And so, I don't know. I mean, I think about this all the time. I feel like, because of that, this kind of, like, issue around, you know, my passive, you know, sort of data trail and the extent to which I'm being surveilled. And so, you know, partly I can have this conversation with you around AI and feel some optimism and enthusiasm about the types of opportunities the technology might create. And yet part of my mind always defaults back to these questions about, you know, how do we avoid this not becoming effectively a kind of authoritarian at best, totalitarian at worst kind of situation where not only always under surveillance, but it's very difficult to hide the enormous data plumes that we're all giving off as we interact with these systems every day.
- [38:33]
A
No, I worry about that digital exhaust, like, all the time. So I did this thought experiment a while ago. I wrote about it, but I thought about this a while ago. If you ask a regular person, regular people don't even think about this. But if you ask a person who's thought about it slightly, what do you think is the most surveilled city in the world? Some of them might say London. London famously has all these CCD cameras that the police have access to, etc. But if you actually do the math it might be San Francisco. Like because if you think about ring ring doorbells, every Tesla has something like seven cameras on it. Waymos have more cameras than that on it. We have cruises, we have ray bans with cameras, et cetera. Like San Francisco probably has more camera sensors per square block than any place in the world right now. And what makes it even crazier is, is that all that data is sitting on private company servers. It's not even sitting in the governmental server. So we can argue whether that's good or bad. But I think it's crazy in some way shape or form. So you know, I think that I think about this world all the time of just like are we racing towards some more digital surveillance? Which is why I like to think a lot about also architecture and permission models. Like I am worried that the OpenAI product that Jony I've is working on, I have no idea what it is but might be some form of like surveillance pin or something that's also recording all recording things. I am worried that the new Apple AirPods are rumored to have IR cameras on them so they can have better environmental sensing is what they call it. So I am worried about these type of things. Like to the point that I've jokingly a few times tried to start a company to build anti surveillance technology of just like what would it be to walk around with speakers that are doing ultrasonic clicks all the time so microphones get jammed? Or what would it look like to have jewelry that had high intensity IR LEDs embedded in them just to like blast away any camera that's trying to record my wife when she's wearing the jewelry, like kind of thing. So I do think it's something we need to be really mindful of that we don't want to get into that surveillance state. So the you're. I mean that is another one of the policy angles that we need to be carefully looking at. Like I don't buy the original argument that we have no, we should have no expectation for privacy in public spaces. I think that that argument was long before this explosion of computer sensing devices that we've embedded in the environment. So we really need to rethink what that looks like and what my right to be not only to be forgotten, but what my right is to like that real time privacy. I mean I generally have to go into every single zoom call these days assuming it's recorded and it's Just like a horrible way to live. And I think we as humans deserve better. So I don't have an answer to you, Justin, except just to like complain with you, but I, I agree.
- [41:26]
C
You're reminding me of another guest we had on the podcast a few months ago, Helen Nissenbaum, you know, who of course, great scholar on all matters privacy. But you know, she started imagining, along with her collaborators, you know, many years ago, these ideas about how we potentially cloak ourselves or otherwise interfere with surveillance systems and generally try to carve back the public in a way.
- [41:53]
A
But I do want to get in the world of like victim blaming. I mean, I feel like I would create anti surveillance tech for my wife because I find it amusing and hilarious and the fact that I want to rage against the machine a little bit. But we shouldn't have to. We have not, as engineers been given the appropriate set of challenges in the world. Right. So I feel like it is possible smarter people than me could either agree or disagree, but I think it's possible to build these systems and not destroy human privacy, human agency in the process. But the thing is, the people who are usually funding this work don't care, or they're incentivized not to care in the case of things like Meta or others. And so like, I think if engineers given the right constraints, could still figure out the right systems to do. And that's like one of the things that I'm working on inside Mozilla is just like, how do we introduce these constraints so that smart engineers can architect systems like Signal was able to do it, you can have two way messaging between people that no one in the middle can snoop on. So like, they created an architecture where it's possible. We need to set the same type of constraints for engineers as they're thinking about their work and they get architected, they can figure it out, they're just not being asked to do it.
- [43:16]
C
Right now I'm trying to think about where to land this, knowing that the hour's almost up. Is there, is there, is there a direction that you were hoping this conversation would go where you could, you could finish on knowing that you're speaking to a policy curious audience.
- [43:30]
A
I mean, I think one of the biggest things that we need to go do, both as technologists and as policymakers, is that I think the lay people, and I mean that with utmost respect, just have this view of technology right now that it's either inevitable in the sense that like they're just going to get whatever Apple or Google gives them, or it's screwing them over in some way, shape or form. And that's screwing them over. Population is growth.
- [43:56]
C
It is screwing them over, right? I mean, we're getting screwed over.
- [43:58]
A
Yeah, I didn't, I didn't mean to say that in the way that I thought I disagreed with them. I do agree with them. And look, I wouldn't be working at Mozilla if I didn't agree with those people, like, but I think what we need to go do is we need to educate more people about what these true trade offs are on technology. And we need to incentivize ways to get more alternatives out there into the world. I want more signals, I want more firefoxes, but I want them across more domains of the world. And we need more of these credible alternatives. So in the sense that we can use policy to fund those or can use policy to incentivize those to exist, or we can start funneling more capital or more attention into open source alternatives that might be better or for human agency or better for human, for human privacy or better for human well being, we should do that. Like, those are the things that we need to look at. Like, I want to look at policy as an enabler for these credible alternatives that happen. So for anyone in the audience that wants to think about that type of stuff or wants to work with it, ring me up. Like, that's the kind of stuff that I want. I want more than just Mozilla thinking about these problems. I want an entire vibrant tech ecosystem thinking about these problems.
- [45:17]
C
Is that the source of your optimism, ultimately, that you see this as, you know, a set of decisions that, that people and institutions are taking and we can make different decisions.
- [45:28]
A
One of my experiences that I found lovely as part of doing technically optimistic my podcast was that a lot of times I would get notes from reader or listeners who would say something like, I never thought about it that way before and now I can't unsee it. And so, like, if we can do that more, I believe that we as humans can then figure out the next step. And so like I. My entire career at this point has been a little bit like, can we educate people just to see the tech world in a slightly different way that might cause them to have action later? And so, yeah, I do think that's the source of my optimism that like, when you show people what's truly going on, they're like, oh, that's not good, we should do something about it. I'm like, yes, great, welcome to the club. Now I'm gonna go talk to someone else.
- [46:21]
C
Ravi Kakorian where are the best places to follow your work these days?
- [46:25]
A
So if you just look for owners, not renters on substack, that's probably where you can find me best.
- [46:29]
C
I really appreciate you talking to me about this. I hope we can pick up these topics around agents and MCP and the policy implications of those. That feels like to me a lot of, you know, new new ground that we'll need to cover here.
- [46:42]
A
Well, Justin, such a huge fan of the work and what you're doing there, so anything I can do to help, just let me know.
- [47:00]
B
That's it for this episode. I hope you'll send your feedback. You can write to me at Justin Net techpolicy Press. Thanks to my guest, thanks to my co founder Brian Jones, and thank you for listening.
- [47:18]
A
Tech policy press.