Loading summary
Transcript97 lines
- [00:00]
Dr. Rob Johnston
Foreign.
- [00:04]
Santia Ruiz
Hi, I'm Santia Ruiz, and you're listening to Statecraft. Today we're joined by Dr. Rob Johnston, an intelligence community veteran and an author of the cult classic Analytic Culture in the US Intelligence Community, a book so influential it's been required reading at darpa. First and foremost, Johnston is an ethnographer. His focus in that book is how analysts actually produce intelligence analysis, the ways it works, the ways it doesn't. Johnson answers a lot of questions I've had for a while about intelligence and spying. Questions like why do we seem to get big predictions wrong so consistently? And why can't the CIA find analysts who speak the language of the country they're analyzing? And would your average better on polymarket be a better CIA analyst than the pros more broadly? There's also this meta question that I always come back to on Statecraft. Is being good at this stuff an art or a science? And by this stuff, in this case, we're talking about intelligence analysis. But I think that the question generalizes across policymaking. Would more formalizing and systematizing make our spies better? Would it make our diplomats better? Would it make our EPA bureaucrats better? Or would it lead to more bureaucracy, more paper, worse outcomes? How do you build processes in the government that actually make you better at your job? As a reminder, the full transcript can be found at www.statecraft.pub. if you like this episode, leave a comment and rate us on your podcast platform. Dr. Rob Johnston, thank you for coming on Statecraft.
- [01:35]
Dr. Rob Johnston
It's my pleasure. Thank you for having me. I really appreciate it.
- [01:38]
Santia Ruiz
I'm really excited to get into this, and I want to start with just a very simple question. What's wrong with American intelligence analysis today?
- [01:46]
Dr. Rob Johnston
That's an interesting question. I'm not sure that there's as much wrong as there once was, and I think what is wrong might be wrong in new and interesting ways. What has always stymied analysis is cognitive models, whatever our mental models are of how the world operates and the variables that matter within that world. And that's always influenced by the individual life experience and expertise of the analyst when you're benchmarking a nation. When I say that, I mean something like the CIA's World Factbook, right? Right. Just this basic encyclopedia of essential knowledge about a country we all kind of agree on. This is the truth on the ground as we understand it. And then you have these individual differences that show up and they start digging into these intelligence questions. And the questions range from, you know, wither China, which is so broad as to be almost meaningless all the way down to, okay, here's this weapon platform. Do we know if this weapon platform is at this location or if it's been moved to that location? And do we know if these factories are associated with that, that kind of, that level of detail? The Wither China questions are always driven by poor tasking. Really, the specific in the weeds questions are generally driven by very concrete requirements at a specific time. I don't think the problem is really all of the cognitive effort that goes into that, nor even all of the differences. And I'm a big believer in cognitive diversity, frame it that way, not as ethnicity or race or as diversity in any of those senses, but rather the cognitive diversity that you encounter when you have a bunch of people from different life experiences and different disciplines talking about and trying to solve a problem that if you don't have that, you're missing something. And I see that in engineering all the time. The biggest problem is communicating my two cents. The biggest problem is communicating with policymakers. Policymakers have remarkably short, and I don't mean this in a sense, but remarkably short attention spans. And they're conditioned by a couple of things that the intelligence community can't control. They want to know, is X, Y or Z going to blow up or not? Okay, that's fine. However, if you say yes, probably in 10 years, there's nothing a US policymaker can do about, oh God, 10 years from now. I can't think about 10 years from now. I've got to worry about my next election. If you say, oh, by the way, you've got 24 hours, they think, oh God, I can't do anything about it. It's too late. I don't have a lever to pull to effect change in 24 hours. So there's always this timing, teaming problem, right? If I give a policymaker two, three weeks, that's sort of optimal space for the policymaker. But a lot of the consumers of intelligence aren't savvy enough consumers to know that they should ask for that in the next three weeks. Lay out the three different trends that might occur in country X and tell me what the signposts are for each of those so that I can make some adjustments based on ground truth. So if we see X occur, it indicates that there's greater probability that Y will occur versus A to B. I think that communication between the consumer and the producer really needs a lot of focus and a lot of work. In my experience, it's always an intelligence failure and a policy success. It's never a policy failure. The first person to get thrown under the bus is the intelligence community.
- [05:14]
Santia Ruiz
Will you just go back and define a couple terms for me? Teaming and tasking. So tasking, as I understand it, is I'm the consumer of some product from the intelligence community, and I ask you for it, I task you with it, right?
- [05:25]
Dr. Rob Johnston
Generally speaking, yes. I would differentiate that. Customers are the American people. They pay the bills. As intelligence professionals, we need to think about the customers. They're the U.S. citizens. Our consumers are policymakers, and those policymakers are either civilian or military. And they're tasking. There's kind of a couple of different ways. I can't speak to the current administration because I don't know how they're handling this at all. In the past, there has been the President's intelligence priorities. So every president has a list of these are the 10 things I really care about. And the community puts together the National Intelligence Priority framework. And it says, okay, community, in all the world of threat and risk, what do we really care about? And they lay that out, and it's usually there's some hard targets. Russia, China, North Korea, Iran, kind of usual suspects. All right, let's take those two lists and put them together. And then we will resource collection and analysis based on the merging of those lists. It's a fairly rational, albeit slow process to arrive at some agreed upon destination over the next year, two years, four years, whatever it happens to be.
- [06:33]
Santia Ruiz
And just to clarify, the product of that workflow is we're going to staff this question with a bunch of people and this question with fewer people, and we're going to build our workflows based on that set of lists.
- [06:44]
Dr. Rob Johnston
That's exactly right. That leaves certain things at risk. A good example of that is the Arab Spring. So if you imagine the Arab Spring and you think about a protester self immolating in Tunisia, the number of analysts really focused on Tunisia at that moment in time was. Was minimal. Very small.
- [07:03]
Santia Ruiz
Give me a ballpark estimate, like, how many analysts would have been thinking about Tunisia week to week?
- [07:08]
Dr. Rob Johnston
I mean, honestly, within the community, maybe dozen. In fact, at my old shop, it was like half of one fte for a period of time. Tunisia's not high on the list of U.S. concerns. It hasn't reached that place.
- [07:24]
Santia Ruiz
Okay, when you say your old shop, do you mean the CIA?
- [07:26]
Dr. Rob Johnston
Yeah, yeah. The issue is that that is a trigger for a greater event like the Arab Spring. And when that happens, it's affectionately referred to as cleanup on aisle eight, which is there's some Crisis, and then we have to surge a bunch of people to that crisis for some period of time. Organizations try to plan so that they can staff around crises knowing that we're not going to catch everything because we don't have the resources or the personnel or the programs to catch everything. We don't have global coverage per se. Human is slow and it's meticulous and it's specific. So if you're going to dedicate human resources to something, it's usually a big something.
- [08:06]
Santia Ruiz
And sorry again, but human is human intelligence.
- [08:09]
Dr. Rob Johnston
Human intelligence, yeah. So spies, we're going to go out. Case officer is going to go out and find some spies to help us with collection on Country X. Doing that is, as you can imagine, a very long and methodical process. We may not have resources in Tunisia at any given time because it just isn't high on our list. And then we get a black swan event, right? We get out of nowhere, immolation triggers a whole bunch of protests, and then Mubarak falls. And so you have an administration who says, you didn't tell us Mubarak was going to fall. And the response is, we've been telling you for 10 years that Mubarak is going to fall. The Economist has been telling you for 10 years. Everybody on earth knows that Mubarak can't stay in power based on his power structure. Right? But we can't predict what day it's going to happen. We would love to. That'd be awesome. But realistically, we all recognize that Mubarak is very weak and is hanging on by threads. So the right tipping point he's going to go, but the notion that somehow the community missed it is fictitious. That's generally a policy utterance, you know. Oh, the community missed it, sure. Not really.
- [09:23]
Santia Ruiz
So you mentioned this problem that you just don't have eyes on this stuff that's off the list until you have a Cleanup on Aisle 8. Is a feature of the way that these priorities get put together? Are there other blind spots or weaknesses as a result of that process, that way of setting priorities?
- [09:40]
Dr. Rob Johnston
There are other weak spots because of that. I think the biggest misconception about the community and the CIA in particular, is that it's a big organization. It really isn't. When you think about kind of overstuffed bureaucracies with layers and layers, you're not describing the CIA. You're describing other organizations, but you're not describing the CIA. It is a very small outfit relative to everybody else in the community.
- [10:06]
Santia Ruiz
Can you put some numbers on that? Both the CIA and then other folks. No, you can't. That one's.
- [10:12]
Dr. Rob Johnston
That one's classified. I imagine the. Probably the closest. I don't even know who's a good analog. I don't know how many analysts the FBI has, but it can't be that many. I don't know how we compare in that. In that space. Sorry, I'm not even sure what the real number is today.
- [10:26]
Santia Ruiz
You wrote this great study called Analytic culture in the US intelligence community back in 2005. It's an ethnographic study of the intelligence community. It's assigned reading at DARPA, among some other places. Obviously that's 20 years ago, so things have changed. But I'm curious, what did you notice then that you believe is still true now? What did you identify as features of Analytic Culture that are maybe not true today?
- [10:50]
Dr. Rob Johnston
A couple of things that have happened that I know of since then, since the book. One is it opened the door to research into the community that had not been open before. As a result, a lot of social scientists have had an opportunity to do studies in intelligence organizations all over the world. And it ushered in a chance to do actual research, you know, get your hands on data and actually see what's going on. It was a unique moment in time, but it's translated into a lot more research since then.
- [11:18]
Santia Ruiz
You can't tell me the exact size of the CIA, but it's relatively small compared to the other comparable organizations and the asks that we place on it. So I guess given those facts and the facts that we've got a list of priorities and not everything is on there, how should the CIA or the intelligence community broadly be preparing for these kinds of black swan events? What could it do to do a better job of prediction and analysis?
- [11:44]
Dr. Rob Johnston
I think there's an opportunity to employ a lot of technology for global coverage that we are not yet using or that I'm not aware of are using, let's say. I suspect that places like Polymarket and some of those environments are interesting training ground for agentic AI or interesting training ground for an analyst. Alert that if you see the betting market changing for some reason, it might not be a bad idea to look at Polymarket. It might not be a bad idea to check that out as an analyst. Right. You can broaden your scope of data consumption by going outside of traditional pipelines. There has always been a problem, a difficulty doing that because of the environment itself. But I think that one thing has changed in the most positive way, and that's the utilization of open source Using open source was a tough task 20 years ago. People doubted its value when compared to expensive technical collection. They assumed that billions of dollars on a satellite would solve a problem that it could not solve, when in fact open source was clearly a better avenue into knowledge. One of the things that strikes me in that change over time, over 20 years, is that the very first public reporting about open source, the bin Laden raid, was Twitter, and it was people in Abbottabad who were witnessing this and tweeting live that this was happening. That's remarkably different in the land of media than it was before 9, 11. And so those struggles have sort of persisted. This notion of how good can it be? It's free, how good can it be? It's out in the world and everybody knows it. Well, that's only true. Ish with cyber and the growth of cyber, that started to take on a different role. And I think that that's helped move the community towards a more agnostic information gathering process than it used to have.
- [13:47]
Santia Ruiz
Of the people who we've talked to in the past on this show about the intelligence community intelligence gathering, we've had Edward Lutwalk on. We had Laura Thomas, who was CIA chief of base in Afghanistan. We had Jason Matheny, who I believe you know, who was head of iarpa, now head of the Rand Corporation. There's a couple different diagnoses of things that intelligence community could do better. And I don't think they're necessarily intentioned, but I want to kind of float both of them to you and you tell me what you make of the two of them. One, which Laura Thomas in Afghanistan and Edward Lutwalk, longtime intelligence community observer, said in various ways was that we don't have enough area or geographic expertise. We don't have enough language speakers in specific places. We don't have enough people who understand the places that they're trying to understand and analyze. Laura Thomas talked about in Afghanistan, Walk talked about how in several Central Asian embassies, the state and the intelligence folks there didn't speak the language other than Russian, and they spoke Russian badly. That's one model, and it might kind of connect to your cognitive diversity point. Then there's another model which is Jason Matheny talked a lot about how he tried to build prediction markets within the intelligence community, this successful trial, and then it got shuttered. But the idea was we need to build better systems internally for rewarding predictive accuracy, and we don't do that very well. And instead we reward getting into the President's intelligence briefing or Other short term kinds of products. Those aren't necessarily intention. I think both could be true. But I'm curious, what do you think of each of those theses?
- [15:21]
Dr. Rob Johnston
I tend to agree with the notion of talent. We have a talent issue and the talent issue is confluence of security and secrecy requirements and suitability for the community work and our ability to clear people to come in to work in the community. And it's also a reluctance to engage folks that seem dissimilar from us, as is Rob the anthropologist. That's just sort of a normal human thing, right? The other is always almost scary at first. So you've got to get comfortable with that. But if you can set aside the language of whatever it is that makes you uncomfortable about diversity for a minute, the real thesis is that you really want different cognitive perspectives, all operating together to try to work through a problem. And if there's conflict, that's okay. As long as you manage the conflict, right? It's okay to have agreement, disagreement. It's okay to have heartfelt discussions about whatever it is that you're working on. All that's okay. You just have to manage for that. That should be expected. The problem is if you never get there, right? So if the entire recruitment process and selection process, security process, weeds out anybody that doesn't look like me or doesn't have an education, doesn't have a doctorate, or doesn't have, you know, something like that, that's a problem. So we need to address that. I wouldn't be surprised if we could find clever ways to air gap the most secret information and the most secret work that we're doing from the less secret work that we're doing so that those two can come together in some way. I think that's reasonable and I don't think it's a bad assumption. I do have some issues with predictions and prediction markets. Okay? So the biggest problem I have is that the research that exists indicates that the best way to get good at predicting is to keep score. You've got to keep track of it. You've got to write down, literally write down what your prediction is, how you made that prediction, and then the outcome of that prediction. And there's a bunch of reasons for this. The most glaring reason is that humans in retrospect always think they did great, when in reality, you know, the truth is humans can be better or worse at that task. I don't disagree with the notion that we should be helping people become better predictors. I think that it is a little misguided in the community. And so a lot of people that are focused on point prediction have missed the fact that the community often has to show up and say, all right, here's the scenario. And if these variables or signposts are hit, scenario is going to spin that way. It's a lot more like a hurricane weather map than it is betting. Right? So polymarket is great because it gives you a flash insight into how the trends are going and it's the trends that are important right in this instance. But it's not so much the point prediction, it's really the do we understand the full swath of the path of the hurricane and when will we know which direction it's going to go? Those are a lot easier issues to deal with in some ways, but they require a lot more thought and patience and time and data than gambling. And so I think the reluctance always was a gambling market makes some sense. But betting on one point prediction is not really what we do. So how do we find a better solution there?
- [18:46]
Santia Ruiz
How does the intelligence community do the fundamentally useful thing that you're talking about of assessing its predictions in retrospect and doing what, you know, some folks would call a hot wash, you know, an after action review of predictions you made? I mean, if it's harder to do it in a literal, did we make money on this single point prediction or not? What is the intelligence community's model of doing?
- [19:07]
Dr. Rob Johnston
I had the great fortune of starting the Lessons Learned program at CIA and running that for some time and then running the one at DNI as well. DNI being the Director of National Intelligence. Sorry. And that was a first attempt to get the agency and the community writ large to buy into the notion of, you know, the army does this after every exercise. They go to Fort Irwin, they run a simulation, they sit down, they do a hot wash at the end of the exercise, and they wake up and do it all over again. It's not an unusual thing for the services. Army has a big center, Navy, they all do. And the idea was, let's get the community to buy into that process. The hot wash process, part of it was centralized and part of it was, look, just do this in your office, right? You don't need to involve us. You can just do it on your own. You want to involve the Lessons Learned program, okay. Or you could just do it.
- [19:58]
Santia Ruiz
But the point is just build the habit of looking back at your predictions and how you got to them and reassessing.
- [20:04]
Dr. Rob Johnston
And that can be true in, you know, the forecasts that we do are not just Analytic forecasts, They're forecasted everything, right? I mean, I've got an operational plan. And so in my operational plan, there are forecasts about, oh, is it safe to go here? Am I being followed? Blah, blah, blah, blah, blah. All that kind of stuff that's all baked in. So when you're making those kind of decisions and those kind of predictions, sure, why not? It's universal. In that case.
- [20:27]
Santia Ruiz
And before you launched the Lessons Learned Program, did this stuff not exist at all? Was it just not formalized?
- [20:33]
Dr. Rob Johnston
There has been a history program at the agency since the 50s. They have had a staff historian, but the history time requirements are different than lessons learned. So the historians would prefer to look at the missile gap in the 70s or in the 60s. Right. They would like to think about that frame of time. What happened yesterday after we did this thing is less likely to show up on their radar. And so that gap is where the Lessons Learned Program was trying to focus. Somewhere between the old historical records and the archives and today's operations and analysis, they were doing it ad hoc and without sort of a routine that the military uses. And so this was formalization, but it was also capabilities that were a little bit different than mostly the lessons learned folks were social scientists, not historians, although there were some historians. But our methodology was different and our process was different and our product was different.
- [21:30]
Santia Ruiz
What kinds of lessons were consistently learned in the Lessons Learned Program? What are the themes that come out of that review?
- [21:37]
Dr. Rob Johnston
There's an argument that lessons learned are more accurately described as lessons collected or lessons archived rather than learned.
- [21:45]
Santia Ruiz
Because learning institutionally is hard.
- [21:47]
Dr. Rob Johnston
Because learning institutionally is hard. And not only is it hard to do, but it's also hard to measure and it's hard to affect. There is that kind of cop out, which is, well, we're not really learning. We're just, okay, fine. But realistically, if nothing else, practitioners became more thoughtful about the profession of intelligence. To me, that was really important. There's a lot of fiction about what it's like to be in the CIA. I mean, the CIA is well represented by lots and lots of fiction from Archer to Jason.
- [22:19]
Santia Ruiz
I was just watching one of the Bourne movies thinking about that.
- [22:22]
Dr. Rob Johnston
So all of that is good for the brand in that sense. Whether it's nefarious or not, it's always good for the brand. I mean, at the very least, it scares our adversaries, but it's super far removed from reality. So reality looks just about as dull as reality in general does. It's being a really good financial analyst Being a really good business analyst, any of those kind of tasks. They're all pretty similar tasks. They're all working a certain part of your brain that you can either train and improve or ignore and just hope for the best.
- [22:54]
Santia Ruiz
I don't think any of any of those are dull, but I take your point about the perception versus reality. Yes.
- [23:00]
Dr. Rob Johnston
I don't mean to suggest those are dull, but generally speaking, they don't run around killing assassins. So, you know, it's a lot less of that.
- [23:08]
Santia Ruiz
What do American intelligence analysts do if they're not doing the fun stuff from the Bourne movies?
- [23:14]
Dr. Rob Johnston
They read, they think, they write, they write some more, they edit, they get told their writing sucks, they go back, they start over again. Some manager looks at it and says, is this the best you can write? And they say no, they hand it back to them and off they go to write it again. It is as much of a grind as any other analytic gig. Right? You're reading, you're thinking, you're following trends, you're looking for key variables. And analysts who are good on their account generally are analysts that have picked up very specific tips and tricks that they may not even be able to articulate. One of the things that I found challenging was that the people who are the best performers in the agency had a very difficult time explaining how it was they went about their analysis. Right. They had a hard time articulating their expertise. That's not unusual. Experts really aren't very good at articulating why they're experts or how they're experts. But we do find that after an N of 10,000 ish cases, they get better because they're learning what to look for and what not to look for. That comes with some penalties. So the more hyper focused you are on topic X, the less likely you are to think that topic Y is going to impact it. And often it's topic Y that comes in orthogonally and just makes chaos. So how do you create expert novice teams was a question that we struggled with. Finding the right balance between the old hands and the new ones. You wanted the depth of expertise along with the breadth of being a novice. They would try anything because nobody told them they couldn't. And that's a very valuable thing to learn from if you're an analyst or if you're an analytic manager is how to balance that structure.
- [24:57]
Santia Ruiz
In 2005, you wrote this book, Analytic Culture in the U.S. intelligence Community. And it's this ethnographic study of the intelligence community. It's quite successful internally. I mean, it's assigned reading at darpa, among other places. And it's, as I understand it, a spin out of you setting up the lessons learned program. And in there you really focus on how much of this expertise is implicit. It's not formalized. It's pretty hard to formalize. But you also push for. We should be just much more conscious about how we pass down knowledge that currently this is you in 2005. Most of the information learned, the tips and tricks are just learned on the job. There's not a ton of active mentorship. How do you think about that balance when people are pretty bad at explaining how I got good at this thing and you typically learn by shadowing people and by failing a lot. How are you supposed to try and formalize that sort of thing in intelligence work if people don't even have a clear sense themselves of why they're good or not?
- [25:50]
Dr. Rob Johnston
There's the reality as it exists now, which is you're right, people aren't very good at that. People don't describe it well. They don't even really understand it very well themselves. As an expert, ideally, what I had hoped for 20 years ago was technology to shadow the analysts. So you would have today what would be agentic AI, you would have a digital twin that would consume information that would be aligned with your decision criteria as you make decisions. That it would be trained on you doing your day job, but it will also be trained on you doing exercises and training. And so the algorithm would create a fine tuned model of your decision criteria and you'd have to go back and adjust it. It's not permanent. Eventually there'll be overfitting and eventually, you know, all those kinds of problems.
- [26:36]
Santia Ruiz
Sure.
- [26:37]
Dr. Rob Johnston
But instead of making lessons learned or after action or expertise the thing you do at the end of your job, why don't we make it the thing that you do while you're doing your job? That didn't exist then and it does now. And so today I would probably be advocating for something more like using large language models to help create digital twin like entities for analysts as they're learning their job. There are lots of security issues. And yes, I would fully expect both it and security to go nuts. And I'm sure that that would take probably years of negotiation. But I don't see why not. I mean, I don't see why we couldn't get there. And that strikes me as a particularly useful place to get to because those problems of expertise and articulation of expertise haven't changed. That's still a fundamental issue with human cognition. As an anthropologist, I might be able to do ethnography. I mean, I can go and interview 3, 400 people. I can live in the community for a couple of years and get immersed. Now I'm bilingual, I speak their language, blah, blah, blah. But that doesn't give me the kind of coverage that a digital shadow would give me. Digital shadow. And I'm making that term up. I don't know what it means. Yeah, sure, but that. That algorithm that becomes Rob, Algorithm one is really tuned to trying to figure out how I think about the world so that when I'm not there either, I've been run over by a bus, I've retired, I'm sick, whatever it happens to be, I'm just not present. And somebody wants to tap into whatever elements of my decision criteria they're interested in, they should be able to. I mean, that's kind of the ideal. But knowledge management had a long way to go 20 years ago to get here. Just the idea of knowledge graphs and agentic AI and large language models, all of that stuff has the potential to have a huge impact on knowledge work generally.
- [28:26]
Santia Ruiz
I'm really interested in this question. And something like a year ago I talked to a guy who worked at the State Department on this. We had him on the podcast, Stan Spakogny, and he had a similar perspective on the way State does its own learning. They're not working so much with classified information, but they have a lot of these same questions. You're trying to figure out what's going to happen in a country. And their information, their memo system is really kind of hopelessly arcane. There's no codification of how information comes in or out. There's not really any systems for assessing. Was this desk chief especially good at predicting? Especially bad. But it's interesting because I had some of the same questions for him as I think I do for you, which are about what are the limits of building information management tools and how much can you improve efficacy by treating it like a science. In both that conversation and in this one, what we ended up saying is you can definitely be more consistent and more rigorous about data entry and trying to map and pull from it without having to go so far as to say, like, if we built a digital version of it, it'd be much better than our human analysts.
- [29:33]
Dr. Rob Johnston
I'm not sure that a digital version would be better than our analysts. I don't look at AI as a replacement technology for genuinely complex cognitive tasks. It doesn't pass threshold for me. It is really Good for a number of other things. There are issues about hallucinations and again, overfitting and all of that. I teach a class at Hopkins. Right now I'm teaching 1 on AI in the intelligence community. And in that class we talk about what the perils are with AI, that if you understand what the real guardrails are for our use case, it will help you figure out good ways to implement it. But ideally, if I had any authority whatsoever, I would argue that we should probably offload as much transactional work as we can into automation and free up more time for thinking. And the problem is that it's hard to sell that to anybody. I mean, we say, okay, well what if we can cut two hours out of every analyst day just getting rid of the transactional work? That's two more hours. They could be thinking deeply about this problem or they could be reading, they could be learning, they could be whatever. They could be learning the language, they could be in countries, you know, getting a first hand experience. Lots of things they could be doing instead of filling out form. The problem is when you go to get your budget and you tell them we're going to liberate two hours for people to think, you get blank stares. That's swell, but does that mean I'm going to get more reports? You know, no, not necessarily. Does it mean my reports will come in new colors? No, why would it come in new colors? You know, it's just sort of where is the thing? And so you go to the cissi, the Senate Select Committee on Intelligence, or the House version, the HPSI Permanent Select Committee on Intelligence, and you say, we're going to free up time for analysts to spend more time thinking. And in their heads they're like, oh, great, what do we have to show for that? You know, how can we demonstrate that two hours more of thinking is going to change our ability to deal with crises, or our ability to make forecasts, or our ability to negotiate. You know, how do we know that there's payoff there? Well, principally because there are ways to measure it. There are KPIs, but they're not flashy, they're not sexy, they're not, you know, you're not going to roll into DARPA and say, hey, give me a gazillion dollars because I've got this great idea to give people time to think. Yawn. You know, that's not what people want to hear.
- [31:57]
Santia Ruiz
In this book from 2005 about analytic culture in the intelligence community, we talk a lot about the kinds of cognitive biases that are an issue in intelligence. And they'll be pretty familiar, I think, to most, even lay listeners. Like one of the ones you focus on a lot is just confirmation bias. Can you talk a little bit about. I'm an intelligence analyst and I'm working on some tasking. How does confirmation bias show up?
- [32:21]
Dr. Rob Johnston
It can be subtle and it can be not so subtle. The subtle version is when I come to a new desk, I come to a new analytic topic. You know, I've rotated jobs. It's been a couple of years now I'm off to whatever the next gig is. I'm still an analyst. What I'm going to do when I get to that account is I'm going to read in. I'm going to read all the stuff that has been produced around that account before I got there. That'll give me a good idea of what the community has been saying for the last two years at least, or more likely. And then we call that the analytic line. Here's the narrative for the last 10 years around this analysis. If we deviate a lot from that narrative, we're going to have to have very concrete reasons why we deviate, because that's been consistent finding across people over time. The subtle problem is every time I read into that, I'm conditioning myself to think about the problem in the way that we have been thinking about the problem for the last 10 years or however long. And the more I do that, the more I fall into the same traps that everybody else has fallen into. I'm now consuming all the things that they've produced. And that is having a subtle or not so subtle effect on my thinking about the account. A great example of this. This if you're in dc, China is an adversary. And if you're in San Francisco, China is an opportunity. Those two worldviews, both of them make sense in the context that they are in. But if you read into your account in dc, China will always be the adversary. You're never looking for other reasons for China's behavior other than adversarial behavior, which means you shut the door to all sorts of other stuff. And conversely, if you're in Silicon Valley and all it is is an opportunity to make money, Right? That's the big thing you may not be thinking about, well, are they going to steal our ip? Are they going to get into our finances? Are they going to use our tech as a backbone into other companies? Are we exposed? Because our SCADA systems are exposed. Right. And so you're not thinking enough about the threat Part you're not thinking enough about, the opportunity. Part you're missing that we get conditioned to that.
- [34:22]
Santia Ruiz
This seems like a tricky problem to solve because most of the time, for most topics, on most occasions, I'm guessing you do want it to follow the party line institutionally on whatever the account is. And most of the time, getting read in is making you better. The problem is that the intelligence community, fairly or not, is judged on the times that are not like most times, right?
- [34:44]
Dr. Rob Johnston
That's very true. If we took the notion of shorting the market every day, eventually we're going to make a lot of money, right? There's someday we will make a lot of money. Up until that day, we're going to lose a lot of money. We could do analysis like that. But the problem is what happens for the other 364 days, right? You have to have a lot of money to play that game, or in the case of the agency or the community, you have to have a lot of people, a lot of resources to play that game. And if you have a constrained budget and a constrained workforce and a constrained pipeline between the people who are onboarding and the people who are retiring, you don't have the capital to play the black swan game. You don't. If we want the community to be more attuned to that, it will take considerably more resources than the community has or has ever had. And generally speaking, those resources are driven towards collection platforms, not towards humans.
- [35:39]
Santia Ruiz
Tell me about that. My impression has been the trend over time in the intelligence community has been human intelligence. It's hard, it's costly. Sometimes China gets all our spies and kills them. Signals intelligence is expensive, but it's relatively risk free compared to putting people in foreign countries. And as a result, the trend has been more signals intelligence, more exquisite satellites, fewer people on the ground, language acquisition is less of a priority. Is that kind of mental model that I have in my head roughly? Right. And if it's not, why not?
- [36:13]
Dr. Rob Johnston
No, I think it is. I think you're correct. I think the problem is that it's a bit of a false choice. And not on your part, but on the community's part. The community thinks if we have signals intelligence, if we can get a conversation between two parties of influence within a country, that conversation is going to be more revealing than the human. But that assumes that the people in that conversation are. Don't just lie to each other all the time. A great example of that is Saddam Hussein and the weapons of mass destruction. Kevin woods wrote a book called the Iraqi Perspectives Project where he went to Iraq after the U.S. got there. And Saddam had recorded all of his meetings. And so he had access to all of the recordings and all of the transcripts and all the content from those senior military Saddam interactions. And in every single one of them that he could find, Saddam's generals were lying to Saddam because they all wanted to live. Right. He would say, how's our nuclear program? And the generals would say, oh, it's going great, boss. Or how's our Chembio program? Oh, our Chembio program's just. It's out of control. It's wonderful. We're doing great work. It'll scare everybody. None of that is factually accurate. I mean, it just wasn't true. If all we're doing is listening in on conversations like that, we're going to be just as bamboozled as Saddam was. And I fear that when we think about, oh, technical collection gets us truth. I mean, technical collection gets us access truth. I don't know. I have seen pretty mixed findings. I think the budget impetus. If I'm going to build a satellite and I can engage satellite subcontractors in 43 states versus hiring a thousand more analysts, what do you think I'm going to do? I'm a politician, right? The thousand analysts aren't going to come from my district, you know, so I'm going to go for the satellite. McGuffin, whatever. It happens to be in a former.
- [38:10]
Santia Ruiz
Job a couple years ago, I talked to quite a few NASA contractors for reporting stories and it was always remarkable how explicit they were that the reason you should work with them is they hire in all 50 states and they hire in your state and they hire in your colleague's state, and it's just explicitly part of the cell.
- [38:26]
Dr. Rob Johnston
Yeah. And that is true. I believe that to be true in most places, anywhere that we have to rely on policy du jour that will win out. So unlike China, where we don't have a long plan for a lot of stuff in this country. So what do we do instead? We try to at least balance our spending so it's more representative for district to district. Plus, we want those people to vote in favor of it. So they're going to vote in favor of it if we can help their district. This is where politics come in. Unfortunately, it's not always aligned with mission. Right. I mean, it just isn't. There's a different agenda there.
- [38:59]
Santia Ruiz
I trying to find a way to ask this question that won't get me slapped down with, yeah, that's classified I bet you can answer a version of this. What characterizes really good analysts or really good desks? You know, groups of analysts at the CIA or in the intelligence community broadly. What do we know about the folks who really bat above average?
- [39:18]
Dr. Rob Johnston
The people who are really good are the people who understand sourcing and how important sourcing is for critical thinking. I see this in students and I've seen this all over the place. The broader question about education and preparation for the career. Right. The education itself should be focused on helping people recognize and refute bullshit. That's sort of step one. Is the critical thinking necessary to say, well, this makes no sense, or this is just fluff? The people who are really good at understanding the quality of a source, understanding the history of that source, understanding the source's access to information, or lack of, and the people who are professionally trained to do that are librarians and have been for years. And librarians have a certain way of thinking in the context of the library. But those skills about data quality are really important skills, and we should probably steal shamelessly from librarians when it comes to thinking about this. Data journalism, same thing. There are lots of parallel professions where we could be learning more to improve our own performance. The folks that I've seen who crush it, folks who are really good are the people that. They're like a dog with a bone, you know, they will not let go. They got a question, they are going to answer the question if it kills them and everybody else around them. Right. This is a kamikaze thing. They're going to figure this out. Those people, the tenacious ones who care about sources and have critical thinking skills, or at least tools to help them think critically, seem to me the highest performers, to me, as a rule, they all keep score.
- [40:53]
Santia Ruiz
Really?
- [40:53]
Dr. Rob Johnston
Yeah. It's sort of part of their process is keeping score. So it doesn't strike me as odd that they're really good at it. That's the distinction I've seen.
- [41:02]
Santia Ruiz
There's a dynamic that happens in lots of different professional fields where if you're really good at the kind of basic level, at the analytical level, you get moved into management. I was reading a book called Ghetto side by Jill Leovi, which talks about detective work in LA and in Compton, and really dramatic, how the people who are really good at finding murderers are taken off the streets by their own success, that you move upstairs and you stop doing the thing that you're really good at. I'm guessing that happens in the CIA. I'm curious, like, to what degree is that something that the intelligence community thinks about and tries to prevent against.
- [41:38]
Dr. Rob Johnston
It's a perennial problem that if you succeed wildly, you're going to get promoted. And the problem is that the very skills that you have that make you a super analyst may be skills that'll make you a terrible manager. There's no indicator that those skills will transfer. So that's something that I think people have to think about. The problem is incentivizing and providing a career path for the people who are super at analytic tasks anywhere in the government. There's the senior executive service, right? So you go from the general service scale from 1 to 15, and then after GS15, you go into the senior executive scale. How do we get analysts on a path to make it to a senior level? When their day job is analysis? We don't want them managing anything. We want them focused.
- [42:27]
Santia Ruiz
And you ideally would just like to pay them a lot more money, right, to do that work that would be more consistently.
- [42:33]
Dr. Rob Johnston
Best case scenario, the people that have managed that very well, and probably it's a result of budgetary pressure, is the State Department. INR's analysts are all uniformly. They have really good analysts. INR is the intelligence branch in the State Department. They have long dwelling. You know, their analysts work on an account for years. They might spend their entire career working one account. That's unusual and that's not replicated everywhere in the community. Probably should be. There probably should be a cadre of senior analysts who work an account their entire career because they love the account, they love solving those problems. They're really good at analysis.
- [43:12]
Santia Ruiz
A problem that I often hear about in the State Department Foreign Service, in the folks who staff embassies, is they get rotated every couple years. And that rotation is formally because the State wants to avoid Santi Ruiz getting really attached to the nation of Hungary that he's working in or Slovenia or whatever, and developing some sort of dual loyalty problem or a bias that clouds judgment. Is that why people move accounts at the CIA? Is that the same reason that you're shuffled around? Is that not the right model?
- [43:43]
Dr. Rob Johnston
I'm sure that plays in the back of some people's minds when they think about this, but in my experience, the notion is if we treat analysis as its own fundamental skill, the going commentary was they can be an analyst on any account, they just need time. I don't think that that's entirely accurate, but that was sort of the running narrative. The idea of rotating every two or three years was to help you broaden your horizons, to help you become a Better generalist. Out in the world of analytics, if you start an account that's, oh, I don't know, Honduras, and then you work Honduras for a couple of years and you go on to the next account, it's maybe a little more important on the international stage, maybe it's a little more dangerous, a little more at risk. Diplomacy is a higher stakes game and on and on and on. And then eventually you wind up in management, et cetera. So it is the path to management to be a really good generalist. And most of the promotions, if you want to make senior service, most of the promotions are management promotions. They're not practitioner promotion. The carrot that we've dangled is if you want to get promoted, you got to be a generalist. You got to work a lot of accounts, you got to demonstrate your chops in a bunch of different places, which is great for the career, but not so great for the account. If you get your first tour person for every tour and you're an account that turns out to be important, you never have dwell on that topic.
- [45:07]
Santia Ruiz
That country or that region wasn't important. And it was always this place. You stick folks first to get their feet wet.
- [45:13]
Dr. Rob Johnston
Yeah, exactly.
- [45:14]
Santia Ruiz
Will you define a little more precisely for me accounts like, is an account a country? Can it be a region, a topic?
- [45:21]
Dr. Rob Johnston
Sure, it can be any of those. It can be a functional area like weapons of mass destruction or weapons proliferation or crime and narcotics, et cetera. It can be a region, it can be Southeast Asia, it could be South Asia, it could be et cetera. It could be a country account. It can be Tunisia. Those accounts are broken up and they fit within the various centers and directorates, such as they are in the organization, based on either a regional view or a functional view. And so what we were doing when I retired was trying to realign the directorate style of management, which was managed around career services, and integrate them into centers, which was managed around topics. And the notion was that if, you know, get somebody into a center, they can work on a bunch of different accounts, but it's all related topics and that you don't lose the expertise as they move and move up in the system. You're keeping that within the organization. I left before we saw what the outcome of that was. So I don't know what the outcome of that was, but I would hope that the outcome was positive. But it's just hope. So you show up and you're like, man, I am passionate about Latin America. Give me a Latin America account.
- [46:36]
Santia Ruiz
How did you know?
- [46:37]
Dr. Rob Johnston
Yeah, so Just here's a guess, right? I care about it. All right, well, let's start you out with Venezuela, right? It's a hot topic. They've got a natural resource. Let's put you on that desk, you'll get some sense. Plus it's enough to keep you busy because it's a busy account. And then in time you're like, well, the center of gravity is really Brazil, for my interests, right? I want to go work on the Brazil account. Well, that's great, but you got to go learn Portuguese because you only speak Spanish. You go to a language learning rotation, you pick up Portuguese, you go back to that account, you work on that account for a while. If we had all of Latin America in one mission center, you could do that without the mission center losing what you know about the account. Because if I've read into the account, I know that you may have authored X number of papers that I just read. So I can call you or walk down to your office and say, hey, this just happened. Did you ever see this? Did you ever confront this problem? Or what do you think this means? The capacity to do that is part of the appeal of putting people together in that situation. The organization's knowledge hasn't evaporated because I know that you're down the hall, right?
- [47:44]
Santia Ruiz
What was the model before that? I might be rotated more willy nilly.
- [47:49]
Dr. Rob Johnston
The model before that was career services and directorates. So if you were in the directorate of operations, you were in operations. And so you may go to Latin America and your next tour will be Eastern Europe and your next tour will be Africa and your next tour will be Asia, all over the place. And so it's the tribal knowledge that you would get when you co locate often is lost. And so you would hope to rotate those people into positions where they're training the next cadre. Sometimes it works, sometimes it doesn't, sometimes it just doesn't work out. But it's a lot more happenstance in that model than purposeful.
- [48:27]
Santia Ruiz
That old model seems more James Bondi, you know, the character goes more places for the movie at the cost of a lot of effectiveness.
- [48:35]
Dr. Rob Johnston
The consistent problem is that the effectiveness measures are poorly articulated and poorly understood by both the consumers and the customers. And so the best consumer of intelligence that I have ever interacted with was Colin Powell. And Colin Powell had a very simple truism. Tell me what you know, tell me what you don't know, then tell me what you think in that order so that I can parse out what you're saying. And make sense of it. He was a remarkably savvy consumer of intelligence. Not all consumers are that savvy. Many of them would benefit from spending a little time learning more about the community, understanding a little bit more about the relationship with their briefers, relationship with the analysts. And the more engaged the policymakers are in learning about intelligence, the more savvy they're going to get as consumer. Until then, you know you're throwing something over the transom and hoping for the best. And it's not a great way to operate if you have consumers who want.
- [49:36]
Santia Ruiz
Your product, who were some relatively poor consumers of intelligence.
- [49:42]
Dr. Rob Johnston
There are so many in this current climate, I couldn't even hazard a guess. Dick Cheney was not a poor consumer of intelligence. He just had an agenda and he understood the discipline well enough to exercise that agenda. I would say that Rumsfeld was not good and Wolfowitz was much worse at it than Wolfowitz thought. And there were some others in that administration and I don't mean to pick on them. There were plenty of lousy consumers under Obama and under Clinton, it just is. But not a lot of them take enough time to really think about what they're getting. And the biggest problem that I have found with ambassadors or generals or other consumers is they'll go out into the world and they'll shake hands with their counterpart and they will decide based on that interaction that they understand their counterpart better than anybody else does. I went to lunch with so and so I should know. The problem is, of course, that so and so is not going to tell you the truth. Right. If so and so is going to do something, going to lunch with him probably isn't going to be very revealing. Probably going to tell you what you want to hear. And you'd be surprised how many consumers don't even think about that as a possibility. It boggles my mind. There's a large population.
- [50:58]
Santia Ruiz
It's funny you mentioned Donald Rumsfeld as a poor consumer of information because one of his famous truisms to go back to your Colin Powell mention, was he wanted you to explain what your known knowns and your unknown unknowns. And my first impression would be that's a guy who wants to know what we know and what we don't know before assessing what you think.
- [51:21]
Dr. Rob Johnston
The problem with the Rumsfelds and the Kissingers and some of those guys is that they maybe they are the smartest person in the room, but maybe they should stop believing that for a while and that gets in their way. You get somebody like that in the room. And they just assume from the jump that they're smarter than everybody. Not just everybody individually, but everybody collectively. Now, you can demonstrate that's not true in all sorts of ways. Doesn't matter. There's a certain amount of ego that goes along with all of this. And when the ego gets sufficiently inflated, you reject information that is contrary to your own values, your own mental model, your own thought processes, and you assign a outlier status. Anything that doesn't conform with the way you think about a problem, that's the expertise run amok. That's where the people like Rumsfeld or Kissinger come off the rails, is they just assume, well, I'm smarter than everybody, so I'll figure it out. You just give me raw data. I have not seen a terribly successful model of that before. It's better to walk into a room and assume that you're not remotely the smartest person there. Always go in like that. You're doing yourself a cognitive disservice if you think you're cleverer than everybody else. It's the rookie mistake. But you see it over and over, and if it works for you and you keep getting promoted, eventually you start to believe it.
- [52:42]
Santia Ruiz
Doesn't seem like a rookie mistake to me. It seems like a seasoned professional mistake. You know, it's a mistake that you're more prone to the further on you get.
- [52:48]
Dr. Rob Johnston
That's true. Yes, you're right. You're right. It is a longevity error.
- [52:55]
Santia Ruiz
Rob, last round of rapid fire questions for you. But tomorrow the President comes to you and says, rob, I don't trust my intelligence community folks anymore. I need you to clean up the place, and you've got full reign to make whatever changes you think necessary. You've got political cover from above to fix the intelligence community. What's on your laundry list?
- [53:17]
Dr. Rob Johnston
Well, I'd have to understand what the principal's vision was in part, but also when Denny Blair was the Director of the National Intelligence, he and I had known each other in a previous life, and so we kept a nice relationship. We would interact regularly. We would discuss what are the challenges facing the dni. You know, you are, in theory, the top boss of intelligence. The biggest challenge was budgetary authority didn't go with the job. So the budgets all went out to the agencies. The budgets weren't coordinated, which meant that there were no levers the DNI could pull. The best they could do was make recommendations and hope they implemented a whole bunch of committees to corral all of this into some kind of decision. Making process and that's slow and it's cumbersome and it's difficult and it's not always effective. I would want to see budgetary authority go to whoever's going to be designated as in charge of the community or whoever is at a cabinet level seat. So cabinet level position, budgetary authority, discretion about moving people around. If I'm a good analyst, you know, we have this thing called joint duty and joint duty is a requirement for promotion. You have to go spend a couple years outside of whatever organization you're from. I mean we've seen a lot of positive from that. There are ways to do it that are a lot more fluid. And so I should be able just to reach out and grab a bunch of people who are experts on Latin America because there's a cleanup on aisle eight, Venezuela, right? I go out and I just grab everybody in the community who's working Venezuela, put them down. And I don't have to negotiate barter, haggle, hassle, I can make it happen. Those kind of authorities. It is not very gee whiz, it's just hands on kind of stuff. Our concept of security needs, needs modernization.
- [55:07]
Santia Ruiz
What does that mean?
- [55:09]
Dr. Rob Johnston
Secrets have a shelf life. All secrets have a shelf life. There are no secrets that will last for infinity. It's just not how it works. And the more people who are read into a program, the greater the likelihood that there's going to be a leak, et cetera, et cetera. But instead of recognizing that as ground truth, we pretend that we can hold onto this stuff for 25 years, 50 years, however long we think we can hold onto it. It's a fool's errand. There are secrets that need to be held onto very tightly, but there are only a few. There are lots of secrets that are just like it's a secret this week. But once this happens, there's nothing secret about it. We spend inordinate amount of energy on the it's not going to be secret next week problem. There's a part of me that thinks, you know, we're wasting a lot of energy on this when we could be using that energy in other ways. And if there are 10 things we really have to keep secret, let's keep those 10 things secret. Let's put those resources to use on those areas that are critical rather than everything. It's an unpopular opinion amongst security professionals, by the way.
- [56:16]
Santia Ruiz
Why is that?
- [56:18]
Dr. Rob Johnston
It opens up the possibility of more insider threat. Who's the next Snowden? Who's the next. Yeah, et cetera, et cetera. That kind of thing. Having been outed by Snowden, I can appreciate that. But we're going to have to be real about resources. At some point I have to sit down and say there's just not enough resources to be hyper focused on everything. Got to stick to what really matters.
- [56:42]
Santia Ruiz
Before we close any other unpopular changes to the intelligence community, we're going to roll out in the Johnston regime.
- [56:49]
Dr. Rob Johnston
There has been a long standing argument. Boy, I'm going to get in the weeds and I apologize in advance. Sherman Kent, who is arguably the father of US intelligence analysis, he was the guy in when OSS was formed, he was the guy that was going to be the, the puba of analytics. Eventually it was the Langer brothers. There was a Langer at Harvard who did a analytic background bio of Hitler. It's out in the world. It's really interesting. His other brother, that was Walter, the other brother ran the OSS analytics shop. There was a colleague to Kent's, it was Kendall. And they had a very specific disagreement about intelligence role in policy. Sherman Kent believed completely in the idea that policymaking should be divorced entirely from analysis. Kendall on the other hand, thought that intelligence was political and that if it wasn't involved in the political process, it probably wasn't serving any particular administration. And guy named Jack Davis, who is a very well respected analyst in the community, 50 years doing analysis, he's written about that debate. He's written about a bunch of other stuff. So I recommend Jack Davis's stuff. He's passed away, but he was wicked smart. I would want to tackle the problem of politicization in intelligence, that there are some things, yeah, you should be piped into policy, you should understand policy. But there are some things where policy should not be part of your cognitive process. You need to divorce yourself from policy objectives in order to do analysis. Right. So that balance, it still has room to improve. That's what I'm trying to say.
- [58:30]
Santia Ruiz
Give me a rough sketch of how you improve that. Because my impression of the intelligence community is that at least formally on paper they inhabit that kind of Sherman role of we're just telling you information. And a lot of the criticisms practically, especially over the last eight to ten years have been, okay, you say that, but in practice the job is riddled with political ideology.
- [58:52]
Dr. Rob Johnston
I have seen that played out myself and that was with Iraqi wmd. And I'm more sympathetic to the analysts in Iraqi wmd. That problem issue failure. Because for the most part there was a very stated political desire to invade Iraq and it was driven by People around Bush, and I think personally that that had too much influence on the decisions that were being made in the community at the time.
- [59:25]
Santia Ruiz
I don't think you're the only person.
- [59:27]
Dr. Rob Johnston
To think that that's a problem. And we've seen that problem, that policy can drive you into bad analysis if you let it. We've also seen the problem where bad analysis, and I don't mean to pick on analysts, and I apologize to those who might be offended, but bad analysis, like 9 11, have a lot more to do with cognitive failure in the community than they do with policy. And people will argue, oh, no, that's not true. You. You know, I live through the commissions. Trust me, it's true. It's true enough. Work with me. How do you do something about that? I firmly believe that there has got to be a willingness by policymakers to engage in some kind of boot camp around intelligence. If you could get together policymakers and intelligence practitioners, professionals, and just engage in a regular conversation, they could start to have exchanges that were more grounded in the reality of each other's lives. And that, I think, is a giant disconnect. That since we don't live in policy and they don't live in intelligence, we're always sort of bumping and it's always clunky. There have got to be ways that we can come together and work out a better system, a better process for information provision and information intake. It shouldn't be this difficult, but it is.
- [60:44]
Santia Ruiz
I think that's a great call for us to end on. So policymakers, when you're listening to this conversation, reach out to Dr. Rob Johnston and his colleagues in the intelligence community. We'll put together a little retreat in the woods for all y'. All.
- [60:58]
Dr. Rob Johnston
I would hope that exposure would come in handy. I suspect that it is a problem of human distance.
- [61:04]
Santia Ruiz
I believe it. Well, Dr. Johnston, thank you for joining.
- [61:07]
Dr. Rob Johnston
Thanks for having me. I appreciate it.