Greg Conti (2:46)

How in what ways could we be evil?

Tom Cross (2:48)

Right.

Greg Conti (2:49)

You could imagine a company self assessing for that and coming up with a list of ways that they could be evil again, capabilities that they're aware of, capabilities that they deliberately don't use. Maybe they would discover capabilities they didn't realize they had. And then you could ask a set of questions about that. Maybe there are architectural or political checks and balances we want to place within our company to ensure that we'd never use those capabilities. Right. And so we started talking about in the context of that, the relationship between companies and governments, particularly in times of conflict, and how those capabilities that you may not even realize you have, and you certainly wouldn't imagine using, you might find yourself in a position where you are using them. And so we think it's worthwhile to have that conversation and to imagine those things and to start thinking about what position you want to take and how.

Tom Cross (3:46)

Yeah. And I'd add to that that the end users only see the tip of the iceberg of what are the true capabilities. If you imagine a social networking, you think, oh, I can connect with people and look people up in this directory. But the social networking site knows every direct message ever sent, the day everyone joined, the day, every photograph ever used, every IP address that people connected from. And they have the graph of the entire network. And so they only expose a little tiny fraction. And so many people think that that is the full, you know, the full end state of what that company can do, when in reality it's like 0.01%. So the idea is what are the circumstances where those capabilities would be used by the company? Decision by an insider threat actor, an external threat actor trying to get in or government has ability to Defense Production act to leverage if compel companies to use their capabilities.

Greg Conti (4:50)

Capabilities.

Tom Cross (4:51)

So anyway, it was a good talk. It was good fun.

Greg Conti (4:54)

So the backstory of our talk is that Greg and I gave a talk at a very, we'll say corporate computer security conference, and we had a slide in it that talked about capabilities that companies might use in a military conflict that they don't realize they have. Right.

Greg Conti (5:17)

How might they use the capabilities of their organization in an offensive way midst of a conflict, which they might choose to do depending upon their valence to that conflict. Right. And the conference was very uncomfortable with us having that conversation.

Tom Cross (5:33)

They asked us to remove the slide.

Greg Conti (5:34)

They asked us to remove the slide. And so Greg said, well, okay, we're going to do an entire talk based on that slide and we're going to do it at defcon, where we're allowed.

Tom Cross (5:43)

Yeah, we can have the conversation.

Greg Conti (5:44)

We're allowed to wade into these like, ethically challenging discussions. And I think it's great, like, DEFCON is the right room for these kinds of dialogues. And I, and again, my point is that, you know, that, that they're vital to have. I think it's valuable to, to, to, you know, put on the black hat and look at, look at things from that perspective and Understand that. And then, and then what you choose to do with it is, is, is your decision. Right. And so it's, you know, any tool has both like malicious and beneficial uses.

Interviewer / Host (6:19)

Yeah.

Tom Cross (6:20)

Before your adversaries do the same to you.

Greg Conti (6:23)

So one of the things that we recommended was that, you know, the governments consider this. So we talked about what companies should do, which is something we've discussed. We also talked about what governments should do. And governments, you know, could think about like, what kinds of capabilities exist within companies that could be used in, you know, certain, we'll say evil ways. Right. But then they have to ask, you know, one, maybe they want to use them. Right. But then they have to ask, maybe another state will come in and use them in a way that's not aligned with my strategy. Right. Or maybe the people that run that company will use that capability in a way that's not aligned with my strategy. And this really happens in places where conflicts are occurring. You know, the companies may independently shut off a satellite system. You know, so they're making their own choices that affect, you know, the course of events. Right. And so, you know, looking at all, you have to understand what the capability is to ask those three different questions. And then, you know, what can you do to make sure that that capability is in fact used in a way that's aligned with your strategic objectives and not someone else's.

Interviewer / Host (7:29)

Yeah, that does sound, does sound very interesting. It seems like there's a lot of different ways you can go with that because, I mean, taking any one of those three insider threat, external threat, or government. I don't know if I want to use the word coercion, but.

Tom Cross (7:45)

Commandeering.

Interviewer / Host (7:46)

Yeah, commandeering. Like any one of those could be a good conversation starter.

Greg Conti (7:51)

Yeah. I mean, another thing is that like we've been having these conversations about companies, tech companies in particular, for the past few decades that are like privacy centered. Privacy is the most significant implication of, you know, information technology. Right. But increasingly we're deploying things that, you know, robots, drones, things that can affect the world in a variety of other ways besides just collecting data about it. And those things, you know, have sort of potential negative consequences that are entirely new that we haven't, we haven't thoroughly considered at this point.

Tom Cross (8:30)

Yeah. So the key takeaway for us was it's not what something claims to do, it's what it actually has the capability to do. And we had some fun with an evil robotic vacuum. What could evil robotic vacuum do? Well, it maps your house, of course.

Michael Heller (8:46)

Right.

Tom Cross (8:47)

It could listen to all your Conversations and report back for ideological compliance. It's literally a vacuum. So it could be harvesting DNA. Right. It could be sniffing, it could be war, driving all the short range WI.

Greg Conti (9:01)

Fi in your house, duplicating your access card for your office.

Greg Conti (9:06)

And we found some really interesting vulnerability research that people have done into robot vacuums. And there's some of them don't do a good job deleting data, some of them collect more data than you'd expect. Many of them have like AI image recognition capabilities and some, you know, sometimes there's code in there for doing face recognition. It's like, why is that code even there?

Michael Heller (9:28)

Right.

Greg Conti (9:29)

So, you know, there's, it's, it's. If you think about like what is the ultimate malicious vacuum? I mean, George Orwell could only imagine a television. But you know, we have things that move around and increasingly we're going to have more and more of them over the next like five to 10 years.

Tom Cross (9:44)

And consumers are just going to be stuck at the surface level of what, you know, just, they're going to be reading the marketing copy and parroting that pack. So it's cool. Like the hacker community, there are people who specialize in reverse engineering robotic vacuums for years and years. So like the hacker community has like world class expertise and they the right mindset to think about these things.

Interviewer / Host (10:07)

Yeah. I remember maybe five years ago at defcon there was a, a woman who was researching like social engineering using robots. Like, because if you have a little robot vacuum, it can be kind of cute. You send it into somebody's room with like a little package. You know, it seems unthreatening, but it's an easy way to deliver something that could be dangerous.

Greg Conti (10:36)

Yes, absolutely right. Yeah. We did include intentional destructive malfunction as on the list of capabilities that it could have. Right.

Interviewer / Host (10:45)

And so going through this exercise, like, as what are the main things that you think companies probably are not thinking of that could be used in a malicious way?

Greg Conti (10:59)

The main things that companies are not? Well, I mean, I think it depends entirely on the nature of the company. Right. So it's really about like, should you do the sort of assessment I suggested where you flip the hat and you say we're going to be evil, what are the list of things that we could do? And I think we also, one of the lenses through which we considered this is again, because we spent a lot of time looking at the relationship between companies and governments. Sometimes companies are operating independently in a way that is not aligned with like basically the national strategy of their country. Right. And other times, you know, governments, you know, are commandeering capabilities from companies. Right. So, you know, when a government is going to commandeer a capability, they're going to look systematically through the, through the lens of what they're trying to accomplish. So in a time of war, governments are interested in collecting intelligence. They're interested in influencing public opinion. They're interested in engaging in reconnaissance. They might want to know, you know, what the inside of buildings is like, which the robot vacuum will give them. They're interested in, you know, getting access to networks, Right. So maybe, you know, a wi fi light bulb company might be an interesting way for them to get, you know, sort of access to a network that they otherwise don't have access to. And so you, you know, depending upon the nature of the technology your. Your company creates, you know, that tells us, you know, what, what potential malicious or evil use cases the technology you have could be put to. And then you can assess, you know, again, like. Like a lot of companies assess for maybe customer misuse or they assess for insider unauthorized misuse. Usually they do not assess for misuse by the management team or misuse by a government that commenters the capability. And it's like, are there guardrails, like institutional practices or architectural technical architectures you can put in place that would limit those misuses if you desire to.

Greg Conti (13:00)

So the idea. Idea of the Ulysses pact. So Ulysses wants to sail through an area where there are these sirens, and he doesn't want to be attracted to their song and change course. So he binds himself to the mast of a ship and sends it through that area and, you know, the sirens show up, but he can't change course.

Tom Cross (13:18)

Yeah, and I would add, too, that we found that, like, we teach that we teach a course in adversarial thinking. And so, like, this was kind of an embodiment of that, you know, logical flow for the talk today. But the larger problem about where companies need to reflect on their own capabilities, we found that adversarial thinking is teachable. And we have people cheat in class and cheat on a test and some other things, but at the end of it, they're better tuned. So the idea is companies can better reflect on their own capabilities with the right color hat on, if you will.

Interviewer / Host (13:55)

And I was just about to ask that, is this something that you would teach a red team to do? Who in the company should be responsible for doing this?

Tom Cross (14:07)

Well, so we've taught. I've taught a red team. I ran through the training, but I also worked with a company that had Their own basically, red team services. But they brought in their marketing person, the sales person, development person. And you know what, it helped them all because for the sales people, they understood those, you know, the mindset of information security people, right? Or the mind of the customer, how they're doing. So but at the same time, like early career, mid and early career infosec people, developers for sure, right. Like they're coders. They're not thinking like an adversary. Right.

Greg Conti (14:46)

It comes back to this adversarial way of thinking. So maybe it would help to like, sort of double click on that a little bit and talk about, you know what I mean? And one of the things that we talk about in our adversarial thinking class, like we have some content in there that tries to sort of capture the mindset of the hacker scene. And I think that the mindset of the hacker scene is in sort of like, it's like anthropological. And so, you know.

Greg Conti (15:18)

When you consider the career path of a professional engineer, and I've followed this career path, you go to elementary school and you learn math, and then you go to college and you learn some more sophisticated math, you learn calculus, and then eventually you get to a place where you, you start to do engineering. And by the time you're a senior in an electrical or computer engineering program, maybe you're finally making a computer. So you have to build up all this knowledge to get to the point where you have this artifact.

Greg Conti (15:53)

In my life, I also had the opposite path, which is I'm a kid, I've been given this computer, and I have to go top down in terms of trying to understand it. So I start by playing games. Then maybe I learn how to work with the operating system. And then, you know, maybe I learn how to write software. And, you know, maybe I get into, you know, some of the electronics at some point, you know, so you're coming from the top down and you have the difference in perspective.

Greg Conti (16:20)

You know, that, that, that exists. There is. And this comes back to, I think, a point that Greg made earlier there. An engineer created this artifact, and they did so because they intended it to be a certain thing, right? And as a, as when you're coming from the top down, you don't necessarily have that mental model. You don't know what this thing is supposed to be. You have to discover that, right? And it often turns out that the reality of the artifact is not the same as the intent. And that gap between what the thing really is and what it was supposed to be is the, the space in which A lot of interesting capabilities or security vulnerabilities exist. So I think developing that mindset, I use the word anthropological because it's like, imagine I discovered this thing and it's thousands of years in the future, and I don't have any cultural context for it. I don't know what it's supposed to be. The only way that I can figure out what to do with it is through discovery. And so it's like William Gibson said, the street finds its own uses for things because there are things that are made for particular purposes. But, you know, folks that get them don't necessarily have that cultural context, but they find other things that they can do with them that are relevant to their world.

Tom Cross (17:41)

Right?

Greg Conti (17:42)

So that's what I think is the essence of security vulnerability research. It's like finding, you know, finding the truth about the nature of these artifacts. And I think that if you're really good at thinking that way, then you can take a company's product services and you could ask, what is the truth? What's actually possible? So, for example, let's take an antivirus product. Antivirus products look for files, right? And they're supposed to only look for viruses. But what if I went to an antivirus company and I said, well, this document was leaked. I'm going to give you the hash for the document. I want you to search all your customers to see if any of them have a copy of this document. Right? The antivirus product becomes a surveillance system. So. So the thing is that when you point out something like that, people say you're crazy. Right? You know, this antivirus company is never going to do anything like that. That's insane. Right? And so the point is that that makes. That's true today. Assuming that company is financially motivated and ethical, right? Then we assume that they're not going to misuse the capability that they have. The point that we're making is that, let's say there's a war.

Greg Conti (18:53)

The consequences could change. Somebody could show up with the Defense Production act and say, we're taking over and we need you to do this stuff. And you may want to do this stuff because the circumstances are such that it feels dire.

Interviewer / Host (19:06)

Once you find.

Interviewer / Host (19:09)

These points, these products that can be used in a malicious way, whatever.

Interviewer / Host (19:15)

What then? Like, obviously you can put in policies where you know you're not going to comply with government.

Interviewer / Host (19:28)

You can't put in policies. You have to put in technical capability. You have to remove capability or you have to.

Greg Conti (19:35)

Yeah, so what is the list? Right. So certainly you could remove Capability.

Interviewer / Host (19:39)

Right.

Greg Conti (19:41)

You could to have a technical architecture which makes this thing either difficult to do or which makes it transparent if done right.

Greg Conti (19:53)

There are also, you know, sort of there, I think there are institutional processes. Perhaps it's not possible for you to prevent the institution from deciding to do it, but you could design things in such a way that lots of people in the institution would know if it was being done.

Tom Cross (20:10)

Right.

Greg Conti (20:11)

So that they can't be done, you know, sort of quietly in a corner. And then one of the things that I talked about is like maybe, maybe, you know, a third party NGO could come in and audit and publicly say they're not doing it. And you know, if that relationship were to break down, the organization may not admit that they're doing it now. But it, you know, sort of like creates that assumption. So like there's this concept that, I don't know if you've ever heard of the concept of a warrant canary.

Interviewer / Host (20:42)

Canary.

Greg Conti (20:43)

You know, if you're running a social media site, you might put something out there that says, I've never had to respond to a warrant for which I was, you know, prohibited from disclosing. And then if the warrant canary goes away, we can make certain assumptions. Right.

Tom Cross (20:56)

I've always thought like that's legal, you could do it.

Greg Conti (20:58)

I don't know. Right.

Interviewer / Host (21:01)

We've seen companies do it.

Greg Conti (21:02)

There certainly are warrant canaries out there.

Interviewer / Host (21:04)

Yeah.

Greg Conti (21:04)

Like, you know, maybe, maybe the government tells you you can't take your warrant canary down. And I'm pretty sure that Google, like.

Interviewer / Host (21:13)

As part of Google's transparency report, I'm quite sure I've seen them use canaries before.

Greg Conti (21:18)

Interesting.

Interviewer / Host (21:20)

I would have to go back and double check, but I've definitely seen that in use.

Tom Cross (21:26)

Every company has superpowers. You mentioned industrial control systems. Right. Clearly they have powerful tech that if maliciously used, can be highly impactful. But what we're finding is basically every major company has superpowers. Imagine what a dating site. Just for sake of making this simpler, just think about an evil dating site. What type of data leakage can be collected from that and also at scale.

Greg Conti (21:55)

So I mean, I do think that the practice of information security becomes more and more.

Greg Conti (22:01)

Vital as time goes on. I mean, it's always this question of are we going to maybe we solve the problem. Right. Because we just get really good at coaching developers to write better code. Or we've got.

Greg Conti (22:15)

This whole debate about AI and whether AI generated code is going to have fewer vulnerabilities, which is nonsense. It's got the same number of vulnerabilities because it's reading code that humans wrote and it's writing it in the same way that humans do. And so it's pretty much producing the same volume of vulnerabilities that the humans were. But the, you know, there's this, there's this always been this question. People have been asking this question for years. It's like, are we going to fundamentally address some of these problems in a way that, you know, means that there isn't as much of a need for this kind of work. Right. And I think I'm continually amazed by how this whole conference continues to expand and grow every year and the scale that is functioning at now. Right. You know, DEFCON used to be, you know, like maybe a thousand people in a conference, single conference room in a hotel somewhere. Right. And so, you know, I mean, I think, you know, these issues are going to continue to get more and more complicated. And so I feel like there's a lot more work to do in infosec. And I think, you know, these, we're talking about these robotics, a lot of these embedded systems, like, they don't have the degree of hardening of, you know, some of the sort traditional computers that we use or our phones. You know what I'm saying?

Interviewer / Host (23:35)

Like.

Interviewer / Host (23:37)

The last question is always the same. What is the big takeaway that people should remember from this conversation?

Greg Conti (23:46)

So, yeah, I mean, I think that.

Greg Conti (23:49)

One of the reasons that we're talking to hackers is, is that hackers are good at seeing that distinction between what things are and what they, what they.

Greg Conti (24:00)

What they were, what they were meant to be, and, and figuring out how they can utilize things in ways that were not intended and may not be.

Greg Conti (24:10)

Wanted. Right. And so it's this sort of like adversarial mindset where you, you can, you can think about, you know, what evil is possible within a particular situation that, that can be turned to good. Right. By applying it in situations like this and then seeing what comes out of that application and then thinking through what you want to do with it. And I think a lot of what we do at conferences like defcon and Blackhead is we were willing to wade into these ethically challenging conversations. What if a company was evil? Right. That's a conversation that people are uncomfortable having. And we have that conversation openly. And then, and then we, and then having had it, we're actually, we're actually able to apply what we learned to make the, you know, make things safer. And I, I think, you know, that's necessary.

Michael Heller (25:18)

That's it for today. If you like what you heard, please subscribe wherever you listen and leave us a review on Apple Podcast or Spotify. Your views and feedback really do help us understand what you want to hear about. I want to thank our fearless leader David Moulton, our content and production teams which include Kenny Miller, Joe Betticourt and Virginia Tran. And thanks to to Elliot Peltzman for the mix and the original music. We'll be back next week. Stay curious and keep asking the hard questions. Thanks for listening.

Interviewer / Host (26:10)

It.