B (3:20)

What's a good example of a project that relates to AI for human reasoning?

A (3:24)

Yeah. So I guess I can give a couple of examples and then talk about the. We've kind of mapped the space a little bit more generally and we've got some initiatives that we're trying to get started and we've got some that we're already supporting. So a great example of something already existing here. Have you heard of community notes on Twitter? I can give a kind of rough explanation, but you can think of things like fact checking and adding context. Historically, they've been performed in this kind of centralized or roughly centralized fashion. You've got these media broadcasters, they're relatively few, they're relatively long lived, and they therefore have this kind of reputation and identity. They're sort of trackable by states and so on. And there are some rules they have to follow. Maybe they publish retractions, maybe there are certain kinds of standards they have to follow with regards to their evidence and so on. So this kind of centralized fact checking happened. And then of course, social media kind of blew open that now everyone is a publisher. And so we're in this slightly different world. And for a while people tried to apply the same kind of centralized fact checking mindset. And this wasn't really scalable in practice. Now, Community Notes is one of some kind of innovations in that space. The first principle is can we crowdsource fact checking? And it's not just fact checking, it's adding useful context as things that might, that people might find useful in the context of some kind of social media post. The way Community Notes achieves its kind of trust, because of course, this fact checking is quite a sort of powerful and important duty in a way, or like providing context is really important. People need to be able to trust that. And merely having it be kind of vote based. And this kind of thing, you could imagine that being quite swingy or it could be dominated by some factions or whatever, or manipulable. And so Community Notes achieves this kind of trust. It uses this bridging algorithm, which means it looks for proposed notes that people have written, which achieve some kind of consensus of being useful according to an inferred sort of axis of usual disagreement. So it kind of looks for what that ends up being in practice is that like the principal component of disagreement is usually left, right, politics, but not always. But that kind of gives you this. It gives you this kind of gold standard. For this note is considered useful by a broad segment of society. It bridges gaps. It's useful in that sense. So this is Community Notes. It's crowdsourced in that sense. Now, one project that we're supporting is ways of accelerating that. So a real issue with Community Notes is some misleading or confusing or unhelpful message can kind of make it around the world, go viral in minutes or hours. And the trouble is, this Community Notes process, of course takes time. Someone has to notice and flag this. Then it has to enter kind of a queue or a pool. And the community of people that contribute notes, they have to then notice this, they have to do some research, they have to put together a potential context that they want to provide. Then that has to go into this voting algorithm and we have to kind of infer is it a useful note? And all this kind of thing. So this can take hours or even days. And so that's unfortunate if this thing has already gone around. You know, there's the classic saying, the lie makes its way around the world before the truth, that's got its boots on. So one thing that we're supporting there is can we accelerate that through tools which support note makers and note graders? And one way that you can accelerate that is actually have AI perform some of the research or even drafting of notes, but of course maintaining the gold standard of the voting and the rating and this kind of thing. And there's an interesting kind of design tension there. So that's one direction we're supporting and that comes under the kind of umbrella that we're calling collective epistemics. It's a sort of platform focused intervention there. We could talk about other kinds of interventions there. As another quick example, Another thing we're supporting is various initiatives in scenario planning. You may have come across deep research. This is a kind of form factor that various AI companies have put their products into, where language models have been trained to do not only kind of surface level web search, but also kind of jumping a little bit deeper, doing something like a literature review and then putting that together into some kind of context. So you can see that there's potential there for these quite deeper sort of agentic workflows using LLMs in structured ways with appropriate scaffolding and tuning and so on. And they're able to support these kind of exploratory investigative processes. They're not replacing them. It's not like you would just fire and forget or trust the output entirely, but it's able to inform an analyst or someone like that. And so for scenario planning and this kind of other sort of institutional decision making, and perhaps outside of institutions as well, we think there's a lot of potential there. So we've got a couple of people that we're supporting working on that kind of thing as well. This scenario planning thing is sort of close to my heart because certainly I would have been a customer about a year ago or a bit longer now. When I was working at the AI Safety Institute in the uk, I was just such a kind of person who would have absolutely loved to have this support if it was well made into a quality tool. And so I have several friends who are still working in that context and there are people working in similar contexts in other countries. Yeah. And this really is like elevating, augmenting decision making. And this is another kind of broad umbrella of human reasoning that we think is really important.

B (8:21)

Yes. So one reason why this approach is interesting is because we're interested in keeping people informed and people in the loop while they're making decisions. And so I see both of these examples you just mentioned as trying to empower people to understand what's going on while it's happening. And so not sort of handing off decisions fully to AI, but trying to use AI to empower Ourselves. How do you see that going forward? Do you think that approaches like these can scale as AI scales in the tempo that AI is progressing at?

A (8:58)

Yeah. So this is really thorny and we do give some thought to this. And one way you can look at it is realistically perhaps we should expect in the medium to long term automation will just win out eventually in most cases because it's always going to be more efficient to have the tireless doesn't get sick AI systems that can just kind of run 24, 7 and furthermore they can think thousand times faster and perhaps even they can think better. I think this is definitely on the table as well. But that of course comes with all kinds of costs because once you've taken. I'm not saying anything new here. Once you've taken humans out of the loop, the trust is maybe harder to get. And then also maybe you might think over time you might lose the ability to oversee and understand what's being done. So keeping humans in the loop is one phrase I actually really like, a phrase that Audrey Tang has publicized a little bit, which is why can't we have machines in the human loop? And that is kind of one of the kind of. It's the sort of aesthetic we're going for here, which is there are innumerable ways we can imagine. We already have tools which are part of our individual systems that augment us, part of our collective systems which augment us. Why stop that? Why end that process? We have all these interesting building blocks now. LLMs are the most obvious one and they enable all kinds of design. The design space is massively opened up right now and it's really underexplored and underexploited and yeah, so we think that there's much room to kind of improve that, but it might. Well, maybe you think that this is only a temporary measure. I think that could be the case. But even in that case, there are some really important decisions to be made by society, by individuals, by groups, by societies, about where we actually go next and how it is that we're directing that when we're making various kinds of transitions and which transitions we're making. I think a lot of people think of this kind of linear timeline when a thing's going to happen and then it's just bish, bash bosh, ABC happens. But actually the sequence of things and even the choices we have about kind of what parts of the tech tree we explore, I think these are genuinely open. I don't think it's just a naive to think that there are actually decisions to be made. And so if we can in the meantime enable better decisions to be made by individuals, by societies, and better on their own lights, we're not trying, you know, we're not. The two examples I've given, you can see they're very augmentative, they're not kind of preachy, they're not giving us specific position, but they're supporting reasoning and helping people to come to better decisions. And if we can do that in the meantime, then hopefully, even if it's the case that everything eventually gets handed off to AI, we'll be in a better position to trust that and to know that it's going to be trustworthy. And as a society kind of endorse that and move in that direction in a way that we think is wise. Let's say.

B (11:39)

There'S a lot of talk about chatbots becoming AI agents instead. And, and so how much do you think this is about the form factor or the interface, how you're interacting with the AI models? Do you think that for us to stay informed when we make decisions and make better decisions, do we need something different than a chatbot? And specifically, do we need to avoid agents or do we just need to use agents differently?

A (12:05)

Yeah, this is a good question. Again, I don't think it's a dichotomy necessarily. So you already have chatbots which have tool integrations. Is that an agent? It's kind of an agent. The form factor, the interface, at least the human is still chat. Sometimes there's this whole massive wall of text that you get when it's not doing a good job of keeping you informed about detail. Sometimes you just get this annoying wall of text and that can be overwhelming. Other times you get too little detail, like maybe it's compressing far too little. But the form factor is kind of the same, at least to the human side. But it's doing, it's using tools, it's going in, making web searches and so on. And then there's the kind of, I mean, you can build any kind of tool integrations and we did a lot of this at ac and at that point you're kind of stepping towards something like an agent. But it's not autonomous, it's not living on its own, it's not like out in the world. And that would be like the really extreme end of agency, where maybe this thing is self sustaining in some way, maybe it's even replicating these kind of scenarios can get, I guess, I mean, scary and. Yeah, so moving along that maybe it's not even one axis either, but I'm kind of painting as if it's one axis. Certainly moving along that axis, things get, the stakes get higher, the capacity for oversight goes down and hopefully people are going to make the right trade offs about where to kind of be on that axis. So let's think about deep research. That's kind of an agentic workflow. You're kicking off an agent, it's a kind of contained agent. It has various affordances for search primarily, but also to kind of do sequential search and kind of perform searches on the basis of findings it's already made. You've got various other kinds of like limited agents. People use agents for coding. They often have access to some kind of sandbox code environment. They're able to make edits to files, they're able to maybe pull from APIs, they're able to load libraries and so on. This is another kind of, kind of somewhat contained agent. And then maybe the oversight you have there is in principle you could look at every single commit it's made and every single edit it's made, but in practice you're probably not going to, you're going to look at some kind of summary and this kind of thing. So yeah, moving along that agent axis, at least to a first approximation, it's enabling more to get done between human kind of check ins. But the exact trade off there is that it's kind of reducing oversight potentially. I think there's a lot of design that can be done around enabling the really important pieces of information to be surfaced and also attempting to guarantee that appropriate check in happens at appropriate points as well. Because the more long running tasks, the more the agent's going to kind of run into all kinds of small decisions maybe, but perhaps larger decisions as well. I mean at some stage you, maybe you're imagining things like partially autonomous corporations where you've got various kinds of functions being fulfilled by AI systems. They don't need to be agents as well. I think people imagine drop in remote worker like a phrase that gets thrown around. People imagine this kind of fully fledged self contained agent, but it doesn't need to be. It can be a kind of agentic workflow. It can maintain context over time, it can perhaps learn over time, but it doesn't need to have every single affordance under the sun. And the oversight there can be entirely reasonable if it's well designed. And that's the big crux there. If it's well designed, how do you know it's well designed? I mean there's all these kinds of questions.

B (15:10)

Do you think we are ready to work with the models as they are now? So, for example, we can do a lot of scaffolding, we can implement a lot of tools into a model, but if the model is not aligned with our interests, it might go badly anyway. So you can imagine the model presenting information selectively to us and so pushing us to make a certain decision. Is that something that's potentially solvable with extras on top of the model, scaffolding tools, stuff like that? Or is that a deeper issue that has to be solved before we can begin integrating these models?

A (15:48)

I think that's a great question. So maybe I have a few thoughts about that. One is, yes, absolutely, we should get the guarantees that we need to get about the inclinations of the models that we're putting into these systems to be thorough, to produce legible outputs, to not be biased in certain ways, to not have blind spots and this kind of thing. And a lot of this we might call these are kind of epistemic qualities or virtues. So in fact, one of the pieces of our kind of human reasoning puzzle is can we ensure that the AI components we're putting into these tools for human reasoning, in particular lms, like particularly salient elms, can we ensure that they are virtuous in these ways that we think are necessary not only as kind of conversation partners and as agents, but as building blocks for tools that we might want to build downstream of that? So let's go back to this kind of scenario planning. Deep research, esque, but geared towards scenario planning. Let's go back to that example. If your system has systematic blind spots, or even if it's scheming and it wants to hide certain kind of parts of the mechanics of the world or of the specifics of the situation, which is kind of a much more pernicious thing, then you might expect that it could perhaps surreptitiously or even inadvertently surface a biased summary of the situation that could lead you to systematically biased decisions. I guess I'm trying to think concretely. If there was some kind of political shenanigans going on and there were particular parties that were behaving shadily, you could imagine a sufficiently, either blind spotted or sufficiently scheming model that was part of a system like that, you can imagine it just kind of discarding such references before they kind of bubble up into the ecosystem. And that can give you these blind spots. So yeah, one way we can potentially counteract that is building in these guarantees. And that's a strong Word, I perhaps need to soften that. But building in some level of incentives for the developers and also benchmarking and testing and so on for these epistemic like thoroughness, legibility and these kind of qualities. One way that that gets done in contemporary AI, very often the way that you make progress is by having benchmarks, by having testing suites and this kind of thing. So one way we're hoping to kind of incentivise that whole area is to enable people to build really great environments for testing these epistemically virtuous properties like thoroughness, legibility, skepticism, this kind of thing. There are already kind of examples of that. So you may have heard of retrieval augmented generation, rag. This is a kind of very common way of kind of scaffolding up an lm. You provide a corpus of. Of extra materials. Maybe it's like some materials on a particular domain, those are indexed sort of semantically. And then the model, rather than doing a web search, it can do a kind of search into this index and retrieve content, and then that content enters the model context and then the subsequent kind of answer or generation or whatever it is, can incorporate that. And there are already evaluations that people build, or how grounded are the generations of systems which are using these retrieval augmented components? And so that's like, I can imagine that's a microcosm of this kind of epistemic virtue of being thorough or something like that. And so you can imagine really expanding the way that people are looking at this. And a big part of that is things like being biased or being particularly sensitive to framing or like treating particular people or characters or institutions preferentially or dispreferentially, these kind of virtues, again, you can kind of produce evaluations that at least give some confidence. I'm not going to use the word guarantee again, but at least produce some confidence that the systems, when deployed in other important contexts, are not going to have these kind of pernicious properties. Another thing is, again, look at thinking about RAG retrieval, augmented generation and thinking about web search as a kind of augmentation there, and the importance of legibility of the inputs which have been gathered for a given downstream generation or task or whatever. I think that's an important thing as well, that kind of auditability and traceability, and then knowing that the corpora being drawn from. So whether it's the web search indexes or whether it's these kind of rags indexes, knowing that those are themselves well constructed, well structured, and are able to provide these auditability legibility guarantees that can also enable these systems. So then it's not about the LM itself, it's about the overall system and those other components. But knowing that those things are well constructed can enable ideally more trustworthy downstream system overall. So this gets to another direction that we're kind of considering as well, which is, can we. And this is just kind of one interface into that. But I recently read a blog post with my colleague Ben where we're talking about the concept of a full epistemic stack. And this is where we're asking the questions like what is it that actually constitutes a good epistemic practice? And how does that happen between people? So how does a community have good epistemic practices? And it's things like legibility of citations and sources, what we might call provenance of ideas and data and so on. And if you think about in science, you have some of this, you have citations, but sometimes they're very crude or kind of rough, and then those citations have citations and they make claims. And maybe those claims are kind of backed up by some of the evidence they're pointing to or some of the kind of reasoning or other kind of research they've performed in that piece of work. And then the conclusion gets kind of distorted. You got this, I think in America it's called the telephone game. And Britain we say Chinese whispers. There's this process where the information can kind of get distorted. So enabling that to work better is one thing. And then there's also the process of discourse where a position is put forward, there are reasons to be uncertain or skeptical, and then other investigation is performed, other insights and positions are brought in, and then the kind of the dialogue moves forward and the state of the discourse expands and improves. And enabling that to happen in a more kind of fluent and perhaps audible legible way is important. This is a little bit more kind of nascent. I realize I'm talking kind of various kind of abstractions. I'm waving my hands, which is a sign that I'm maybe not being concrete enough. You can read the blog post if you're interested, but we think so. This is a system where you can imagine building lots of tools and processes, many of which are AI augmented, which enable us to build up these structures and these corporate and these metadata and annotations and so on, which then themselves can be fed into downstream AI systems. So it's using LMS to build the data sets and structures which can enable AIs and other systems to build be kind of more epistemically virtuous downstream.

B (22:16)

So how much do we need to improve on the human side of the full epistemic stack? If we imagine that stack to be a collaboration between AI and humanity, how much of that is on us? So, for example, there's a massive difference between naively using an LLM to try to produce some output and then sort of using best practices, doing everything you can to get the most out of the current models that you can. Do we. Do we need some form of training? Do we need is could this be integrated into the scaffolding in some way such that how, however simple and naive your prompt is, the system will sort of perhaps ask additional questions of you. Some systems already do that. Do we need to improve in order for this system to work?

A (23:04)

I think ideally you're not asking everyone to suddenly become an expert in best practices of using AI, but of course people will learn and adapt and best practices will spread. I guess a slightly different spin on your question when you say, do we need to improve? Yes, absolutely. The whole point of this AI for human reasoning is to enable us to improve in the ways that we might endorse if we were able to coordinate better or if we were able to kind of communicate better. But is it necessary for us to improve in our practices for how we're even using these systems? I think part of the point of things like the full epistemic stack is so the way we're conceiving it at least, is it's meant to be pretty adversarially robust. It's meant to be pretty robust to various kinds of scales of contribution. So whether it's a kind of offhand remark on Twitter or whether it's a full scientific paper, I'm kind of thinking like, ultimately that can be. Ultimately that can be incorporated into the same structure. It's all discourse and it all has some degree of evidence and sourcing and so on. So on Twitter, maybe, the evidence is usually pretty implicit and occasionally people provide links to articles or to papers or whatever when they're making a point, and then that article itself or that paper itself has some context and so on. So there's some structure there. And then scientific papers, I mean, they're a bit better, right? But we've all read papers where we think the citations are, or some of us have written papers where we think the citations could be better. So, yeah, absolutely. I think the way we're conceiving this is can we be as robust as possible to these varying scales of contributions? Adversariality is a really important part of this. So the community Notes example is a case where it's supposed to be as adversarially robust as possible. Because, of course, in a large community of communicating individuals, people have different priorities. Sometimes they compete, and sometimes it benefits me for you to believe something that maybe I don't actually believe. And so, you know, this is deception and this is a huge problem. But we think that. We think an epistemic sac can be constructed which is police largely robust to that kind of thing. Improving practices. Yeah, there is a kind of bias. I've forgotten the name that people use for it, but there's a certain kind of overly trusting approach to using automated systems and computers in general. I don't know how widespread this bias is still today. I do wonder if some of the research done on that may have been older. I don't know what the latest and greatest research on that is. Now, it might be that I would hope at least that people have kind of learned to be a bit more skeptical, but I'm not sure.

B (25:24)

Yeah, it might be increasingly tempting for anyone basically working at a computer to hand over more and more tasks to these systems and perhaps to hand it over in a way that is not fully thought out. So maybe you have to present something to your boss tomorrow and you haven't really prepared anything and so you throw something together using a model. And this is not really how you get the most out of these systems. But again, I'm thinking this is sort of a human weakness. And so is there, are people really trying to get the most out of these systems as they currently exist? Or is the bottleneck perhaps what we are trying to do with them as opposed to what they can become and can do?

A (26:10)

Yeah, that is interesting. So, I mean, you said that it might be tempting to kind of have these lazy uses. I think in some cases it might even be imperative if people are under a lot of pressure. There might even be sort of selection pressure like hiring and firing, and this kind of thing can even come into play. And that can happen at different levels as well. Yeah, I believe in that as a pressure. I think it's really concerning when you say, are people trying to get the best out of these things? I mean, this is the other fascinating thing about LMS is that as people learn how to kind of prompt them and how to tool them up and how best to give them the context they need, it often turns out that a given LM from 6 months ago can actually do some stuff that people had thought it couldn't do, as long as it's provided the right Context and tools. This isn't shocking, but this remains the case even as the latest models come out. Also, scaffolding improvements continue to reveal capabilities that are new. So in some sense people are trying. Could they try harder? Probably. I think maybe what you're asking is, are most people trying or can we expect most people to try hard to get really, I'm going to use the phrase epistemically virtuous again, really epistemically virtuous outputs out of these systems when they're under all these pressures and so on. Now, I don't think it's fair or even sensible to expect everyone to be doing that. But what we can hope to do is enable people to make the best choices about which tools to be using and to be using the tools which are going to equip them best to understand the situation and to make better decisions rather than the tools which are kind of, maybe they're biased or maybe they just aren't helping that person to understand. They're just kind of producing slop or whatever. And the way you enable people to use better tools is both to make the better tools and also to enable people to evaluate which tools are better.

B (27:58)

And perhaps we can then have norms and rules, if you are high up in governments or high up in corporations for how to use these tools best or when they should be used.

A (28:09)

I can imagine that happening. Yeah. Certainly there are already rules for all kinds of things. Like there are already auditing rules for various kinds of high stakes business functions. There are already auditing rules and scrutiny rules for various kinds of. At least in the uk. I'm most familiar with the UK context, but I think it's the case in other countries as well. All kinds of auditing and scrutiny rules for political decision makers as well. So we already have these processes. Yeah. So it's imaginable that that could extend to use of various kinds of AI tools and so on. Whether we can expect or want that to happen on a more broad basis, I think that's perhaps less likely. You can imagine extending some kind of. With kind of epistemic virtue evaluations. We're imagining that primarily applying kind of normative pressure to developers and enabling people to see at an easy glance benchmarks which can kind of tell them, well, this system is the most honest system, this system produces the most legible outputs, this kind of thing. And at that point there's pressure on the developers. You're enabling consumers and users to equip themselves in a way that they're not going to be. They're going to be least misled. And we're not currently imagining that kind of entering into any kind of regulatory regime. You could imagine though, if it turns out best practices and make it relatively easy actually to construct these systems which are on the whole epistemically virtuous, you can imagine it being mandatory or semi mandatory, like, well, why not just use those best practices? Why not actually make sure your systems are kind of honest and legible? It may even not be necessary at that point to regulate. It might just be kind of. Why wouldn't you do that? Of course there are always these kind of long tail of developers, especially open source and fine tuning and this kind of thing, like kind of anyone can make their own system. If they want to make a lying bot, then they can do that. And it's pretty hard to stop them doing that. But people can already go around and lie, so perhaps that's not. They're not going to get a reputation for honesty that way. So maybe that's how we defend ourselves.

B (29:59)

It seems to me that at the moment you can also become dumber by using AI more. You can also. Your output can also become worse. So if you are AI, but it depends on what you're producing, right? If you're producing, if you're a lawyer producing a standard contract, an NDA or something, that's probably that might be doable by models. If you're producing a very high stakes complex contract between billion dollar corporations, that's a different story. And you would get fired for AI generating that. How do you see, sort of is it just the higher the stakes, the more you have to think about whether you're using, whether you should use models and perhaps you should not. But then it seems that the higher the stakes are, the more we should want these tools to help us and we shouldn't just.

A (30:44)

Yeah, yeah, for sure.

B (30:45)

Yeah.

A (30:46)

I think this gets back to the. This gets back to the sort of aesthetic is the wrong word. But the point of AI for human reasoning is to enable and augment human reasoning, to have machines in the human loop, in Audrey Tang's words, and that to maintain those guarantees that people are at least kind of understanding what's going on. But yeah, I guess I want to say a few things about that. One thing that I think is maybe pernicious is there's this kind of process in many industries where junior people come in, they do their time doing these somewhat rote tasks. They're learning by doing this, a kind of apprenticeship. They're developing the skills that they then become more fluent in which enable them to then take the more expansive tasks on and so on. And this is kind of progression of apprenticeship and mastery. And this happens in all kinds of domains and disciplines. And we're now entering the regime where the more seniors, the more experienced people can actually, instead of delegating to this apprentice who's going to take many years or whatever, months to train, they can just delegate some of these tasks to AI. They themselves have been sort of augmented and elevated in this way. They now have an army of apprentices. But now the human apprentices are kind of not no longer on that ladder that's enabling them to develop whatever it is in that discipline which they need to become a master, which can include these skills, these kind of drills almost. But it can also include, to some extent, it can include networking as well, which is just as a part of people's context, isn't it? So that's one kind of pernicious thing here, which is a little bit of a tangent from what you asked, but I think this happens at the societal level as well. And then there's at the individual level where people's skills are kind of eroding. Because there's a feeling I've noticed in myself a few times recently where I'm like, oh, I'm sure that, like this thing I'm trying to write, like, I'm pretty sure, you know, Claude or Gemini, whatever could write this. And there's this kind of urge to maybe go do that. And I've tried it a few times, but for me, I tend to find that producing the actual content, it's still preferable to be primarily doing that myself, even if I'm quite heavily making use of AI tools for kind of giving me extra context and research and so on. There's interesting findings in both directions for coding, actually, and I've used some of these tools myself. I'm a bit behind the times myself, so I'm kind of interested to try the latest and greatest. But certainly what I found was that as a kind of enriched autocomplete, it can be very useful. You can move a lot faster, actually, because you know what it is that you're about to produce. And if you're able to kind of describe that in a comment or if there's some kind of preceding context, which kind of makes it obvious, there's so much code which is kind of boilerplate, you can just kind of tab through that and you basically get something that's about right. It's not the exact thing you were about to write, but you like far fewer keystrokes, you know, and you're sped along your way. That's really useful. And I've tried a few of these kind of more like full hands off vibe Cody things. And I've found that I'm sure there's a skill to develop there for me to be able to do that properly. And I've tended to find the outputs are kind of appealing in certain ways, but they're definitely not the thing I was trying to get at. And then there's a whole load of effort to kind of refine that, like, no, please do this, please do this. And that's kind of effortful. But I'm sure the skilled vibe coders are a little bit better at this. But there's interesting findings in both directions there because you hear there's definitely kind of some research which suggests that and definitely anecdotal evidence people saying they're being accelerated, you know, 10 or 100 times faster, whatever, by use of these coding tools. And then there's been some research that people's perceived velocity is much improved, but so far their actual velocity isn't. I absolutely think in principle, and perhaps quite soon the velocity really will be sped up. It may even be that we're at that stage already. But it's interesting that we're kind of on the cusp there, I think, right now. And this is a case where as a coder, like the more hands off you get, potentially the more you're kind of losing access to the, those kind of low level drills which are kind of keeping you in touch with the process. And I'm sure this, this is on other disciplines as well, but I'm most familiar with coding in that sense. And so the same level I was describing, within a firm or within a society, you've got the kind of apprentices are kind of losing out because they're not able to gain those skills because those are being delegated out to AI. Now you've also got within an individual the kind of, the drills that you're by force, by necessity, you're continuing to kind of carry out these drills in the course of your operations. Those are kind of keeping you sharp to some extent. And then if you're delegating those out, there's a slowly kind of atrophy. I think that kind of works. So for many people, the progress of their career possibly includes kind of moving up into this more managerial role. And if that's the case, you can continue to be quite effective even if your kind of low level coding skills. Atrophy. I think that's kind of actually fine. Like many coding managers, the first year or two of being primarily a manager, they're kind of lamenting, oh, they don't let me write code anymore. I've met those people like that. And then. And they kind of find their feet as a manager and then actually become really effective as, as a consultant, as a, like with this kind of more high level vision. So as an individual it's maybe okay if you've kind of got a, got off the ground, but as a society it's maybe harmful then is this just.

B (35:42)

At a societal level? Is this just sort of another step up in abstraction? That has also happened historically. So I guess people now are worse at doing computation, like doing calculations by hand.

A (35:58)

Mental arithmetic.

B (35:59)

Yeah. Writing very low level code. And so maybe we just, we don't need this entry level training anymore because everyone is now, has now stepped up one level of abstraction because.

A (36:11)

Right. Hardly anyone writes assembly code anymore. Yeah, hardly anyone writes assembly. Very few people write C, you know, and this was at one time considered a high level programming language. So yeah, there's this progression there. It's not, it's kind of. Is it the same or is it analogous in some sense? You could say it's literally the same where now we're programming in English and this is the highest level of all and that's being translated into this. I mean in practice maybe kind of intermediate level programming language like Python, which has kind of again historically been considered a very high level programming language. And then under the hood maybe that's being transpiled into CPython or something and that's being turn into assembly. So there's this kind of stack, maybe who knows what language we'll be programming next. It'll be like gestures or dance or something. So in some sense it's literally the same process and it's got us this far. So maybe we shouldn't be scared of it. I think that's kind of right. English is much squishier though so far. All of these kind of intermediate stages, they have fairly well defined semantics and so on, whereas English is kind of fuzzier. But maybe that just enables more flexibility in another sense. Especially when we're not talking about coding in particular. It's more analogous than literally the same thing. But by building tools which enable us to carry out operations which would have been laborious previously, sometimes much faster or certainly at less expense, we're able to not only operate the same kinds of activities more Quickly, potentially we're able to kind of creatively move beyond and open up new prospects because we're able to access, reason about and actually achieve these kind of lower level activities faster and more fluently and so on, and able to then compose them into kind of bigger things. I've forgotten who said it, but there's some quote on this which is something like society progresses by the number of operations it can perform kind of atomically or something. I'm probably mangling that, but I think there's some cyberneticist or others sort of said this kind of thing, or some early kind of computer pioneer, I think. I believe in that to some extent and certainly for AI, for human reasoning. We're hoping that the systems that we're supporting and building and so on can be tools which enable people to perform important operations more effectively or enable operations at all, but without just sort of handing off entirely and sitting back and not really understanding what's going on. But there's a gradient there, isn't there?

B (38:32)

Yeah. If we think about English as the next programming language or as the next prompting language, as the next way of expressing what you want from these models, it's true that we get that's much less precise, it's much more high level, it's also much more expressive. So you can say much more in English than you can in Python, for example. Is this a potential security issue where what you say in English can be understood in so many different ways that it's difficult to actually produce, it's difficult for the model to produce the kind of code or the kind of legislation or whatever it is you're trying to achieve?

A (39:08)

Yeah, I think so. There's pros and cons here. I do think it means people need to be careful and not just sort of naively adopt whatever thing gets spat out at the end, at least in high stakes situations. The lawmaking case is pretty interesting actually because. Or even various kinds of judgment and decision making, because these things can be repeatable. Software is repeatable in a way which really no other process is, because it can be repeatable, it can be auditable and more transparent in principle than a process which involves a bunch of context, messy context being kind of put into a bunch of decision makers who then go into a room behind closed doors, come to some decision which is then kind of brought out on a stone tablet. This is a very illegible process and in principle more parts of that could be made legible and that might actually be a real benefit to society. Because going back to These kind of functions I mentioned in political scrutiny and auditing, it's going to be really important, like, why is it that people are coming to. Or like, what were the considerations at least, which went into the decision that's being made there and that could be.

B (40:12)

Made more legible, actually, on that note. So if you have a system that comes to that recommends a decision, say it recommends a decision on what to do in a corporation, now you ask that system to explain why it came to that decision. Do you think current models are good at explaining their own decision making or are they sort of confabulating and hallucinating whatever actually led them to that decision?

A (40:38)

Yeah, I think the jury's out on that. I've seen a fair bit of evidence that these kind of post hoc rationalizations are, as they are with humans, often basically confabulated. Even if there's no kind of attempt at deception, it's sometimes hard to have introspective access to what were the reasons that you came to various decisions? And so I don't think that's the best way necessarily to be a decision was made. And then you ask, well, why was that decision made? What's much better is if you can buy structure, make it so that as many as possible of the inputs into that decision are legible and potentially scrutinizable after the fact. And so this looks more like what's the corpus which was being drawn from, for extra context in this decision? What's the reasoning trace preceding the decision? What are the various other kinds of inputs? What's the training data look like? And these kinds of things can be scrutable in many cases. It might be kind of a big data problem, especially if we're talking about training data. But if we're just talking about things like coming to a judicial decision, you can look at what were all the inputs that went in there, and at least that tells you what was salient and kind of in scope for that decision. What you don't know then is what else is being kind of brought into latent context by the training and so on into the model. So there's that gap there. I don't exactly know the state of the art in terms of figuring out which training inputs were most relevant for a given decision or given output. And this kind of thing. I know there's work being done on that. So yeah, there's maybe hope to make this kind of thing a little bit more scrutable as well. But of course it's the same for humans. You can't you know, you can ask a human, why do you come to that decision? But you don't know whether what they're saying is either honest or kind of factual.

B (42:19)

A big question here, and I guess you get asked this a lot. Is there a demand for these tools? Do people actually want to get better at thinking and get better at citing their sources and so on? We could imagine right now a form of social media that works much better than what we have, where whenever you're making a claim, you're asked to support that claim. And you know, but it seems that people are interested in playing other games. And maybe you're not optimizing for finding truth, you're optimizing for what's most interesting, what's most engaging, and so on.

A (42:51)

Yeah.

B (42:52)

Is there a demand problem?

A (42:54)

There is absolutely a demand problem. This is very central. And actually, yeah, so I don't think I've touched on this yet, but definitely that's pretty central to FLF's strategy. Probably the reason I haven't touched on it yet is because among the staff of flf, I'm probably kind of least experienced in dealing with distribution and demand and so on. I appreciate the need and yeah, you know, I have some kind of context there. But yeah, some of my colleagues are really skilled at kind of figuring out who has the burning need, who hasn't recognized it yet, what form factors are going to resonate with people, what workflows already exist which you could integrate with and thereby achieve some distribution and enable people. Now you also ask, is there demand? Yes, there's demand. There's a demand challenge, but there is demand. So it kind of varies depending on the tool and on the context, but certainly for things like. So I was in Parliament recently and I was talking to an MP who was interested in what work I was doing. And one of the things I said was, one of our ambitions is we'd really like to improve institutional decision making capacity. And he winced. He winced. He looked very pained. He thoroughly understood that this was a real need, an urgent need. And so he is an example. He's a predictable example of someone who really knows that there is a need. And if something was sufficiently good and trustworthy, I believe he and people in his reference class would want to use that. I also know my previous self when I worked in the government, I would also have used such tools if they were good and trustworthy. And I know some of my colleagues today would. My former colleagues who still work there would today. So there is this demand. Certainly this is kind of the more kind of institutional context, I'm sure there are other kinds of contexts. So certainly things like investment decisions and this kind of discipline, there's a lot of demand there for tooling as well. Whether we specifically want to differentially enable better investment decisions, I think that's not high on our priority list, but certainly enabling better decision making. I also think there is a more widespread demand for better sense making as a society. I do believe this somewhat tentatively, but I have a fairly strong feeling of this. And of course, I have my own sort of biased and filtered view onto what even society is and who is in society and the people I meet and so on, and that will be pretty biased. But I also get the sense from. I get the sense of there's a kind of a zeitgeist, maybe that's too strong a word, but there's a latent desire and a recognition that we need better sense making. Part of the trouble is, I think people don't diagnose what that means for them and for their communities and so on. But I do think with the right integrated Community Notes is a great example. So this has really taken off and the desire is there and people use it. And in part that's because it's integrated into a workflow that people are already using, namely browsing Twitter, but also other platforms. And this is another thing we're kind of trying to work on is can we get systems like Community Notes adopted in other platforms? And when they're there, people use them and benefit from them and are glad that they're there. So sometimes it's not necessarily a kind of proactive demand, but it's like a recognition of a need, a kind of latent recognition of a need, which then, if the right form factor and workflow integration resonates with that need, people then will go and adopt potentially quite quickly. I don't know. I definitely point to there's definitely some large ish segment of the online population on Twitter at least, who are really hungry for kind of better sense making. And they maybe have their own interpretation of what that means. But if you can produce tools which can enable people who have that hunger to better see through deception, to better gain context on the decisions they need to make, to better gain context on the decisions society needs to make, then there's some demand there. But it is a real problem. So it's absolutely central to any project that we're running and any one that we're supporting is, yeah, like, how are we going to get distribution here in a way that's not sort of in a way that's sustainable and that's actually kind of helping people.

B (47:03)

And how important is trust for that? It seems like if you don't trust a tool, you might not be inclined to use it and depending on where the tool is coming from. So say if, as you know, if a government published a tool for online sense making, a lot of people would be sort of, you know, skeptical of using that to develop their views.

A (47:27)

Yeah, yeah. There's well, one antidote here. Historically has been open source. So if you can see the source code, in principle, of course most people don't actually go and look at the source code at all, but if you can in principle see the source code, or if your friend can in principle go and see the source code, or even if your AI assistant that you kind of trust can go and scrutinize the source code, then you have some more reason to at least believe the thing isn't deliberately manipulating you or doesn't have some kind of backdoor or secret inclination. This is harder, of course, for things like. And by the way, I think it's really unhelpful to use the term open source for ML components, like language models, because they're not really open source, it's more like open. If you publish the weights of something, it's more like publishing a binary or something. It's pretty inscrutable. But yeah, open source is good. And the trouble is, of course, as soon as you've got ML components, it's much harder to scrutinize those. And it definitely has been demonstrated. It's certainly possible to kind of hide biases in them through training or even kind of lightweight fine tuning on top at the end. So this gets a bit harder. So yes, if some institution or some organization publishes an ML component of such a system, even if they make it kind of open source in this sense, which is more like open binary, then that trust problem remains. And this is where we go back to, well, are there ways that we can generalizably evaluate for the kind of non bias honesty, all these kinds of epistemic virtues that you might want from such a component? And then can you trust plugging that into your open source software, which is the scaffold and so on. Maybe the kind of trust is able to kind of be built in this piecemeal way, which adds up to some level of justified trust? But yeah, trust is a difficult problem for the kind of epistemic stack concept that I was very vaguely gesturing at you about that Ben and I recently have written on. One of the ways that we would hope to achieve trust would be through potentially open sourcing of various tools that are kind of feeding into the system. And also another source of trust is kind of participatory input or participatory input in principle. So if you look at Wikipedia, almost no one actually bothers to contribute to Wikipedia. I learned that I'm extremely rare in doing so. But you know that you can in principle go and look at the history of edits, you know, that you can in principle go and look at the conversation about a given page, you know, that can even in principle contribute portions or edits to the page. And so I think that gives a certain amount of trust as well. Although even that, you know, under intense pressure of adversarial pressure, even that kind of participatoryness can apparently not be sufficient to justify trust for some people. So, yeah, there are various ways of building trust and a lot of them don't come through. I think the naive way is just like, oh, well, a nice friendly company created this nice friendly AI, so you trust it. I think this is not worth nothing. But there are so many other ways we can kind of build layers of trust and we should be doing that. And I think a lot of those other ways are neglected. And I've kind of named a few of those.

B (50:30)

Yeah, yeah, Wikipedia is an interesting example. It's sort of the crowning achievement of human knowledge, knowledge creation. But it's also. So on the one hand anyone can participate. On the other hand, Wikipedia as it exists is quite elitist in a sense. There's a hierarchy and people. You need to be a trusted contributor in order for your contributions to actually count. As you mentioned, there's very few people contribute almost all of the content. Is that how you see, sort of. Is that the default state of human sense making, that you have this elite? And it's not actually, it's not actually everyone participating, it's actually almost no one participating. Everyone being a consumer of information rather than a producer of information.

A (51:22)

Yeah, there's been a really interesting sort of trajectory on the Internet where my understanding is that the early Internet was a lot more. The power law was much less sharp with regards to who was a contributor and who was a consumer. Nowadays a lot of people just like they scroll the TikTok, they scroll the Twitter and they're basically consuming and occasionally messaging people or something like that. It is interesting. And yeah, Wikipedia has this pretty sharp power law of contribution and all these things have stack overflow. I'm sure is similar. I think there's some inevitability that, and perhaps even some desirability to that because naively at least, we. We don't necessarily want kind of more democracy in all contexts. Right. Because I don't want to be responsible for contributing to a topic I have no understanding of. Why would I want to do that? And what I want to know is that the people who do understand it are honestly putting forth their best effort at contributing to that. And that's a hard problem because of course then there's a kind of principal agent separation, I. E. The principal who wants to understand something that I don't currently understand. I'm trusting the agent, the current expert, to honestly put forth their description of that thing. And this just is. This just is a tricky problem. And often we balance, often sort of checks and balances like situations and scrutiny and auditability and so on, they enable this kind of a bit more trust of a pool of experts rather than mere single expert. It's harder for a pool of experts to kind of collude to, you know, to forward all of their interests at once, whereas perhaps an individual could kind of distort things subtly in a way which is less traceable. So Wikipedia has this property where at least most pages are contributed to by some number of experts at once. But yeah, it's difficult. I think the power law of contribution is probably here to stay in a big way. It might well be that for many things that the largest contributors may come to be AI systems or outputs which are largely AI generated, and you'd hope that they were well scaffolded and sufficiently scrutable and legible, and they're. That their outputs may be auditable after the fact. You can look at what sources went into this and all this kind of thing. Yeah. It remains the case that even with these steep power laws of contribution, things like Wikipedia are to some extent a gold standard of public sense making. Wikipedia suffers from some of the similar problems I've already identified for what we might call legacy epistemic systems. In that it's slow, it's struggled to kind of update in real time. That's kind of difficult. So for many topics that's absolutely fine. If it's like historical biography or something like that, there's not going to be much new evidence every minute or something, is there? But for other things, it's important to kind of keep up to date. And systems like Wikipedia just can't really do that very well right now. But there are various kinds of incremental ways we can imagine improving that. In fact, one of the other visions we have for an epistemic sac is to be a kind of foundation for an improved Wikipedia, perhaps literally Wikipedia itself, kind of ingesting from the more kind of structured metadata, annotated epistemic structures of citations and evidence and so on, but also discourse. Because Wikipedia at its best on a given topic, it's not meant, it's not necessarily so. Something's kind of done, it's known science. It's like we're confident on the whole that something is the case. Then the Wikipedia page for that is just going to describe that situation and then it might describe some kind of history. So it is an ideal case. The Wikipedia page is making legible the provenance structure. Like how did we get to this knowledge in the first place? There's a bit of history. Maybe it describes some of the characters as well, I don't know. And then it's describing kind of the kind of concluded, a summary of the concluded position on this topic. If it's not a done deal, if it's like an active debate right now, again, an ideal case, the Wikipedia page for that will present a selection, ideally all sides of that debate which have some degree of merit, and why those camps are coming to that position and what their evidence is, and why the other camps are coming to the other positions and so on, and where the remaining uncertainties are and where there's maybe lacking evidence and so on. So in an ideal case that's like Wikipedia type format is able to encompass both like solid science and active debate.

B (55:39)

Yeah. And much of this will probably be written by AI. Will it also be read by AI? So will it also be ingested by my agent that's then presenting it to me in a way that's easier for me to understand or perhaps easier for me to digest quickly.

A (55:57)

Yeah, for sure. So whether it's kind of got this intermediate form of something like a Wikipedia page, or whether actually you've just got the slightly more raw corpus, which is the claims, their evidence, their citations and so on, and the positions and the way that the discourse maps together, there's a bunch of structure there that you can imagine actually ingesting directly into a chatbot interface or directly into a Wikipedia like view. Like, I actually, I tend to quite like reading Wikipedia. It's. It's often got like, it's a nice kind of familiar format. I already mentioned that often I want to see a summary, then I want to see maybe something about how we got to that position. Then I want to see are there competing claims, are there areas of uncertainty and this kind of thing. It's a nice layout and maybe if I said to my chatbot, hey, on topic X, can you give me a Wikipedia like summary of this thing? It could then go and access the. Well today it would have to just do some web search and kind of do its best. But if there are better data annotations available, it could query those things, bring them into context and produce me my kind of customized thing. And maybe it knows that I'm quite good at kind of calculus, but I struggle with economics or something so it can be like, oh well, this topic I'm going to explain a little bit more detail or something like that. And yeah, so absolutely there's many ways you can imagine AI might be a big consumer of epistemic content in the future, but hopefully as a kind of intermediate to humans and helping them understand rather than just kind of, I don't know, milling around and doing their own AI things.

B (57:21)

Can you paint us a picture of sort of the perfect state or the ideal vision for AI for human reasoning? What, what does the future look like in say 10 years if this project really takes off?

A (57:33)

Okay, yeah, that's kind of difficult. So let me first give an outline of one way of thinking about the map of human reasoning. So one kind of concept I often go back to is the OODA loop, which you may be familiar with. The Observe, Orient, Decide act, it's something similar, is kind of raised in all kinds of disciplines, but this is a kind of US military context thing. And this is for. It's a decision making cycle. And you see this in reinforcement learning, you see this in cybernetics, you see this in business cycles as well as well as that kind of rapid decision making cycle that we all have to do in kind of individual decision making moments. We're also going through learning and growth. So it's kind of OODLA L for learning. And I think that encompasses a lot of things. So this is one way I kind of, I often use this lens to kind of break down well, when I say reasoning, what does it actually mean? I mean all of these things. And then humans, we're individuals, we're groups, we're societies. So you kind of got two axes there. This is how I think about the very basics of the map here. You've got your OODLA on one axis and you've got your individuals, your groups, your societies on the other axis. And you can start to ask, well, what are the activities that, what Are the broad clusters of activities that we need to satisfy to be doing these things really well. We've talked a little bit about collective epistemics. I think that kind of inhabits the observe, orient, what we might call the kind of understanding side of the OODLA process. And it's both an individual thing and it's a collective thing. So collective epistemics is actually largely about individuals as a community, of communicating as a community network. Like how is it that we come to make sense of things? So collective epistemics kind of carves off that part of that map. Then there's decision making, which is bringing your rich, enriching your understanding, making decisions that you endorse. And this happens at individual levels primarily, but also small groups and so on. And then a big piece that we haven't talked much about at all actually is coordination, which I think is really essential actually to this human reasoning, vision. And very broadly, coordination encompasses is the collective part. And it's not only understanding and making something like decisions, it's also being able to act together. And this is a whole. This is a whole part of coordination, which is kind of difficult. And at least for collective epistemics, there are various aspects to that. We can imagine improving the way that we understand pieces of information that are kind of coming our way. How do we source those pieces of information? That's a kind of more network kind of question. We can improve the way that we relate to other communicators. How do we understand this person's motives, like do their conflicts of interest and so on. We can improve the way that we're sharing information between ourselves. Like this is kind of network question for decision making. We can imagine improving our abilities to model the world, to understand systems, to forecast, to predict, to understand the consequences of what we might decide to do. We can also imagine improving our ability to reflect and understand what even are our preferences. And then coordination. Certainly, I think when I used to think about coordination, I think I used to overemphasize the kind of commitment challenges. I think In Game Theory 101, this is often presented as like, how do you credibly commit to the good thing? I think it's a really important aspect and there's probably a lot of potential for AI to help with that. And we could talk about that. But I think there's other aspects to coordination which are, at least by previous me, were not really thought about sufficiently. And there are other researchers who are kind of far further along that journey than me. Other aspects of coordination include just finding the right people that you need to be in contact with or you need to recognize as a counterparty in some kind of interaction. So we have kind of primitive versions of this now, just word of mouth. People tell each other about their contacts, who they might want to know, people introduce each other to friends and so on. We have LinkedIn now, which is I think possibly maybe the kind of frontier of what technological society has mustered in terms of this kind of networking thing. But it's really important. And one of the kind of high stakes coordination questions I think is how can larger communities and societies coordinate to better navigate a confusing and risky future with technology? And a lot of this might, it won't necessarily fully come down to, but it might really benefit from being able to connect the right people together. And I can 100% imagine ways for AI systems to do that in a much better way than we are able to today. Already the Internet kind of enables this through just web search. If I Google something, maybe I find a blog post that someone wrote and that's really useful and then I'm connected with that person a little bit. But it is crude and it's lossy and it's accidental and so on. So this is kind of networking and connecting. Another really important aspect of coordination is surfacing group wisdom and group will. There are processes like polling, historically focus grouping, these kinds of things where people get together and try to kind of. And even just having a team meeting, this is kind of attempting to surface a kind of group wisdom. I think this struggles to scale. So there's a professor, Jim Fishkin, I think perhaps he's at Stanford, I'm not sure he has a deliberative democracy group. And they point to this trilemma where it's difficult to achieve at once. All of nuanced quality discourse and a kind of political equality and broad participation. You can imagine getting two of those at once, right? So you can imagine getting an expert group together. You can imagine doing a poll. It's widespread, but it's very much not nuanced and so on. And are there ways that we can kind of get more of all three of those things in larger groups? I 100% think there are already actually interesting tools that some of FLF's fellows have created and are going on to kind of continue to create there. I think there's a lot of potential there for incorporating AI to improve how we elicit the group wisdom and the group will of larger groups. This is a really interesting thing. Another aspect of coordination is how do we even reach Negotiated agreement, especially in cases where there are misaligned priorities as well as some aligned incentives. Here, at least naively, you can imagine, minimally you can imagine just throwing a huge amount of reasoning and compute at the problem. Can we simply just discover better options just by reasoning our way there? I think this is kind of. It's not nothing, but I think there's probably much improvement there as well. But this is less clear in my mind. I think the process of negotiation is itself like a very thorny and actually quite kind of complex process. And it's not yet clear to me how to better facilitate that with AI, other than just this kind of naive exploration of possibilities.

B (64:30)

But it might just become much cheaper to negotiate using AI, right? Yeah, it's just much cheaper to send a bunch of messages back and forth and maybe my agent can take my preferences, negotiate on my behalf, and there can be a whole workflow there that I'm not involved in. It will be too costly for me to negotiate directly. But yeah, I could definitely see a world in which we can negotiate over things that are. That now seem way too trivial to negotiate over.

A (64:59)

Exactly. So we can enable more negotiations to happen at scale and so on. I think this is kind of exciting. It's pretty unclear to me what it unlocks. I think in part because it's hard to, you know, this is. We're probably talking about millions or even more of like extra mini micro negotiations happening in principle being unlocked by this kind of capability. And then the question, I don't know, I struggle to imagine millions of things at once. So there's the question, like, what does that really enable in society? I think it enables a certain kind of flourishing. It probably unlocks some kind of economic potential. It maybe also enables better quality of life in certain ways, like just better communities and this sort of thing. It's pretty hard to envisage quite how potent that could be. My rough sense is it doesn't massively elevate sort of human flourishing, but. But I could be wrong. I think there's a chance that it does.

B (65:53)

Why are you skeptical that it might be powerful?

A (65:57)

I think because I. We have kind of limited negotiation capacity and budget and bandwidth and so on. I kind of suspect that at least in societies which are functioning reasonably well, we currently allocate that budget and attention and so on moderately efficiently and are therefore doing the negotiations which are kind of getting us the most bang for the buck. So unless there's that really. Unless the tail of the kind of less salient, lower priority negotiations is really Kind of quite fat and adds up to a huge amount, then I'm skeptical that we can get really transformative change there. This is a rough model, and I don't think I kind of believe it 100%. So there's definitely room in there for it to be really massively transformative. I think some people do believe that, and I wouldn't discourage people from being really excited by that. I think it is potentially really exciting. Another aspect of coordination, which I think I used to neglect, is so we notice that when we are interacting, a huge amount of our interaction is iterated. Even if we're not interacting with the same people over and over again. The world keeps on kind of ticking over like it's a very temporarily extended process we live in. And plus there's this. We've already talked about kind of networking. This is very complex and dynamic network of interacting and communicating people, forming and adapting and growing and shrinking coalitions and groups and so on. This whole complex system produces institutions. It produces constitutions and norms and so on. And what that means is a lot of our. Because coalitions, because constitutions, because institutions and norms are so important for framing future interactions and cooperation and competition, because they're so important, they become themselves an object of interactions and communication. Very often when we're fighting, we're actually fighting over what are the norms by which we should later be coordinating. Perhaps concretely, to make this more concrete, we can imagine pretty high stakes kinds of interactions and negotiations might be themselves about trade agreements, which are meant to then frame a huge number of downstream interactions. So very often interaction is about interaction. Coordination is about coordination. This is kind of property of the complex system we live in. And I think it's also just because humans are that kind of creature. We care about coalitions, we care about institutions. And so because of that, I think it's really worth giving some emphasis when we're asking about coordination, how coordination can be improved, giving some emphasis to how it is that we create and perhaps even design institutions and constitutions and that kind of thing. I think especially as well as new kinds of technological capabilities are entering into the world, there will be questions about how are we managing them, what are the institutions, what are the norms? And this kind of thing, like how is society going to manage theirs, Whether it's at an international level, whether it's a national level, whether it's at smaller community levels. And that will be itself a process of negotiation about institutions and constitutions. And so this to me looks particularly high stakes as well. And I think that there's room for AI to really elevate how we do that, because very often setting up these institutions and the kind of processes we elevate for how we then do coordination and how we apply rules and norms, these things are often at best, they have unintended consequences or they're kind of inefficacious. At worst, they can be captured, they can become corrupted and this kind of thing. Now we as humans have a huge track record of building successful and failed institutions. So we have a lot of data there actually. And I think minimally you can imagine equipping people with some kind of Red Team constitution expert AI which is able to say, oh, have you considered in ancient Greece, there was this time when they tried something like this and it fell horribly flat and they all died. Or oh, have you considered this kind of nearby example where in India they managed this kind of water resource in this way and it was really effective and it's lasted for decades. So there's at least that kind of track record. And then some other interesting capabilities from computation and from AI is potentially being able to simulate and Red Team proactively try to break institution designs and see how corruptible they are, how capturable are they, and try to. And language models actually are able to kind of bring in actually quite semantically rich scrutiny of this kind of thing in a way that we probably wouldn't have been able to do before. This might be somewhat computationally expensive, but I don't think intractably so. And especially for kind of high stakes decisions, you can imagine this actually becoming a kind of move that we make actually. It's like, well, can we Red Team this? How capturable is it? Like, what are the most vulnerable? What are the vulnerabilities we haven't yet seen?

B (70:30)

I mean, one feature of constitutions, as they exist now, is that they resist change. So they, they carry the preferences of past generations into, into our present world. And you know, so you wouldn't be, you wouldn't be allowed to sort of brainstorm a new constitution because the previous constitution is there to prevent you from, from making any, any changes that actually matter. And so unless you're so at, at what level are you imagining that we would be free to design new rules and new constitutions?

A (71:03)

Yeah, that's a really good question. I think I use this word constitution a bit, perhaps a bit loosely, and I should make clearer that I'm doing that. I'm including things like charters for organizations and companies, and I think in constitutions, I'm also including the More flexible parts of lawmaking. And this kind of thing, the word constitution often gets used for this absolutely foundational kind of principles over society. Yes, you're right. The very biggest of these things, but I'm using it a little bit more flexibly than that. But of course, there are these kind of constitutional moments. Amendments get made even to these absolutely foundational documents. We wouldn't necessarily want that to be happening much, much faster. Often these things have lived a long time because they are at least reasonably well crafted, but they do need amending from time to time. So there are constitutional moments even for the most foundational things. Another thing about. I think about this at the societal level, but I think you can imagine it from a team and an organization level as well. One thing about the way that we do institutions right now, there's often quite a tension between agility on the one hand and centralization, or rather agility often comes with centralization in a way that you may not want. You may want the agility, you may fear the centralization because it potentially is capturable or corruptible. And, and so the example, the kinds of case studies I have in mind here is things like authoritarian backslide and coups and this kind of thing. Very often it's not a kind of. You get. There are usually violent kind of aspects to it at one or another stage, but very often it's actually a gradual process of a real or perceived need for agility and executive action, which gets manifested into a relatively small number of people who promise, or perhaps implicitly, perhaps they don't. But, you know, perhaps they promise that, you know, they will lay down these emergency powers when, you know, when it's necessary. And then of course, they never do because the power's been centralized in there. And at that point there's a kind of moral hazard. So even if they were well meaning to begin with, and this is kind of. This is very blue sky and sort of one of the loosest thoughts I have, but I think to myself, well, we have a bunch of new tech. Is there a way that we can apply that to getting at least a bit more of the agility with a bit less of the fearsome parts of centralization of power? I don't know what that looks like. So all the things I've described to you to this point, I have a moderately concrete vision in my head of at least one instantiation of that for this. I kind of don't, but I just have a sense that it's maybe possible, is a little bit utopian. So you Asked me about my vision for, like, what does AR for human reasoning really enable? You know, some years down the line, if it goes really well, I think in. In one sentence, perhaps, and now I'm probably going to accidentally insert a full stop as I speak. But anyway, let's say in one sentence it might look something like a society where it feels like the decisions being made about the important things at the large scale and important things at the local and more small scales are being made with more deliberateness and wisdom and with more alignment and adherence to the people involved and their interests and their needs. I think this is what we want from human reasoning and what we believe is in principle possible. And it's what we look around us and we see is often not the case. We see that instead the world is sort of confusing. It's not clear that anyone really is deciding much of what happens. We wouldn't want it to be the case that individuals or oligarchs were deciding all of what happened, what we would want for affected people, for their interests and principles and values to be reflected in what ends up actually happening. And far too often things are just sort of chaotic and confusing. People fail to coordinate, people fail to understand the consequences of their actions. They even fail to understand what their options even are in the first place. And each of these problems, we think is remediable to some extent with support from tools, many of which would incorporate AI. And we don't think this is going to happen by default necessarily. We don't think the chatbots or the imagined sort of, everyone has a personal agent, all what reason does that have to actually solve this problem? I don't think it does. I think it kind of helps some things, it exacerbates some other things. Whereas with judicious design of tools and technology, which is supporting human reasoning, I think there's a lot more room to imagine that things could get better in that direction.

B (75:42)

What's the best way for listeners to contribute to this project? Where should they begin? How can they see whether they have the interest or the skills necessary to contribute?

A (75:54)

Yeah, so there's aiforhumanreasoning.com there's also FLF.org where you can get in touch with us and you can see some of our other initiatives, which includes the Human Reasoning Fellowship and other work that we were doing in human reasoning. So aiforhumanreasoning.com currently has a summary of the fellowship we ran in the latter part of 2025. So it describes some of the fellows that we hosted, some of the projects that they've worked on and some of the kind of ambitions that they have now. It's likely that we might update that page with other content as we produce it. So that's probably a good homepage for the human reasoning part in particular, if you want to learn more about FLF and our other initiatives and priority areas, you can go to flf.org yeah, I mean, I blog intermittently. I keep thinking I ought to blog more. It's one of these kind of serendipity generators which I think are really important, just putting ideas out there. Earlier I spoke about the kind of the way that the Internet sometimes facilitates connections that you wouldn't have imagined possible previously through things like I stumble across someone's blog post and they mention this thing that sounds really interesting and now I want to go and talk to them about that thing. So I believe in blogging. I sometimes blog. I have recently blogged about one of the slightly more design and engineering heavy parts of human reason that we're imagining with my colleague Ben, and I intend to expect to blog a bit more about that and other things as well. So that's another place you can look.

B (77:19)

And what's the URL for?

A (77:22)

It's my blog is OliverSowerbert. Net.

B (77:25)

Great. Perfect. All right, Ollie, thanks for chatting with me. It's been great.

A (77:30)

Yeah, it was great to be here. Thanks, Gus.