B (7:34)

Hey, thanks, Jim. Thanks for having me. Pleasure to be here.

A (7:37)

First of all, what is Watchtower and what do you do there?

B (7:42)

Really? Good question. We're a series A startup. Ultimately doing what we call preemptive exposure management. Really distills down to understanding if an attacker was targeting an organization, how would they gain access? And so that's what we're doing day in, day out. I'm a principal security researcher and really what that means is that I spend my days looking at organizations trying to figure out how are attackers breaking into organizations and how can we build that back into capabilities to get ahead of the curve, get ahead of attackers and tell organizations you're affected by this thing ahead of an attacker actually utilizing it. And I'm sure it's no surprise to within recent years. I've been pretty busy.

A (8:17)

You certainly stumble into something on this. Stop putting your passwords into random websites. Yes, seriously, you are the problem. Again, I think little in your face on the style, but it's an important subject. So can you tell me a little bit about this? Just give me the bird's eye view of this. You were working through this research. You found these data sets from with JSON files. Tell me a little bit about how you encountered all of this.

B (8:45)

Sure. I suppose it comes back to just curiosity. Right? Understanding. We all use a bunch of tools online every day. Whether it's for convenience or just throwing data in places to make it look different or create a reminder. All these online tools and you always put Information in there. I don't know about you, but in the back of my mind I always have a thing of I wonder where that goes. And it's a question I always like to answer.

A (9:09)

I actually think curiosity is probably what we know that so many attacks have been avoided by somebody looking at something saying that shouldn't, should that be that way? I think it's one of the great skills of security and makes your life more interesting too.

B (9:26)

Very much. Very much. And I suppose this really started with a really simple question which then led into a massive rabbit hole of digging into a bunch of online tools that are really popular with a variety of different organizations, different people, different things, but ultimately led down a path of being more focused on tools that are used by developers, things that they use as part of their day to day workflows that are just kind of part of things they don't think about. If I want to do X, I'm going to use this online tool first and then again it spits out something that's useful and understanding what are the risks associated with that, how can it go wrong and understanding how that can impact organization. And so really that's the focus.

A (10:03)

And in this you found, I guess we'll go through some of those both secrets and passwords, things like that in GitHub repositories, postman workspaces, Docker Hub containers. You found a lot of stuff in a lot of places. And if I got it right, you were pursuing really a couple of tools, JSON Formatter and Code Beautify. That was where you started out in this, right?

B (10:29)

Correct, yeah, those are the two tools that we really focused on. There are some kind of, I would say the popular ones that we all know that should mind makers are Those things like GitHub repositories, postman workspaces, Docker Hub containers, things that are, I would say relatively well known in the industry. We have a bunch of startups that are dedicated to, to purely finding sensitive information accidentally uploaded to them. We have a bunch of tools that are designed to detect it, but ultimately there is far more than just the headline makers. And so really that is what we're digging into. And so this research focused specifically on online code formatters that were being used by organizations that really haven't had that much attention. But as soon as I saw it I thought this is interesting, a tool that developers use to format code, make it look prettier, make it easier to read. They're throwing sensitive code in this and for the sake of convenience probably aren't Going through the steps to remove the sensitive names, remove the sensitive password secrets and things like that.

A (11:23)

So this is again, these tools just really neaten up your code and but in the midst of all that, and the list is enormous of compromised things that you found. Active directory credentials, code repository keys, database credentials, cloud environment keys, private keys, card payment gateway. I could just go on and on. I'll put some of these list of these in the show notes so that people can see them. But it's just what didn't you find is the question.

B (11:51)

Yeah, I think that's a good question. Ultimately, anything that you probably shouldn't be putting into a tool, if it's public.

A (11:59)

Tracing, we found organizations like banks, insurance companies, cybersecurity companies. You got a couple of examples. We'll go through some of those. They'll be anonymous, but we'll probably ask you some questions about those. And in primarily in these repositories, but other places as well. What draw the link for me from these tools. How did you discover these tools and say, oh my God, I can find all these secrets with them?

B (12:26)

Yeah, it's a good question. Like I say, I feel like the industry knows the risks behind GitHub repositories leaking secrets. We see headlines about it all the time. I'd say developers now have a sixth chance to know. I shouldn't sense to them these tools. However, there are far more than just the things we know about. And these are the things that again I was diving into. I, I, me, I'm fascinated with information. I love collecting as much as possible. And I kept seeing these things appear again and again. I kept seeing these tools that are used by developers pop up in my results. When just looking for interesting tools that organizations use and say really? That's what got me digging into them, digging deeper to understand how could you accidentally expose sensitive information.

A (13:06)

But how do you go from that to finding all of these passwords and keys and secrets?

B (13:12)

Yeah, and I suspect many users using these tools weren't doing it intentionally. You can throw some code in, make it look pretty, make it easy to read and then be done. But these tools have a save functionality to save a link and generate something that you could share with a colleague or a friend or someone so they can hit this link and view the code they put in. And by generating those links, that was what inadvertently exposed the censored information. There is a by design page on these formatting tools that expose recent links, recent links that have been saved by the public that you as a researcher or anyone visiting These sites can click on and look at what has everyone else been formatting and saving on these websites.

A (13:52)

But we do this all the time. You do something, you send a link to someone, what's the big deal? He said, knowing the answer. And this blew me away.

B (14:02)

See, that's the issue. These websites had a by design feature to expose the links that have been saved, meaning that not only did I have to have to rely on someone accidentally sending me a link to some code they'd saved, I had a page on these websites that I could go page by page and view. All of the public links have been saved. And in many cases, just like you mentioning, we use these tools all the time. Many users using these tools probably didn't realize when they were generating a link to send to a colleague, to send to a friend that these were inadvertently being exposed and anybody visiting the site could view them because of that. Then don't take those steps to remove the sensitive information or get rid of the stuff that perhaps they wouldn't want in your public facing.

A (14:44)

But on your site there's a huge list and anybody has access to that, correct?

B (14:50)

Yeah, by design feature of the page to show you the recent leaks and the recent things that people have been doing. Almost like a community library, which is a pretty common feature in a lot of tools. I dug into a bunch of things before I settled on code formatters, but generating a public library of things the community are doing or things the community are formatting and saving, it's a common feature. And in many cases, if you didn't realize they were being public facing, people aren't taking the sensitive design.

A (15:15)

So they're saving secrets in these things, posting the links and the links are publicly available. And you encountered this now you've searched through what must be a massive amount of files to find all of these things that are in them, secrets and all of that. And can you tell me a little bit about some of the ones that you found that just amazed you?

B (15:39)

Yeah, sure, it's. But like I say, truly, if you could have a list of things you are meant to put into public tools, we find it. Whether it is from the things that you would expect the sensitive information credentials to log into, tools, databases, web applications, SaaS, platforms, help desks that belong to organizations, things like that, anything you can imagine, we found credentials for which unfortunately the reality is that's pretty commonplace in terms of accidentally leaking secrets. I think the things that concerned me as a researcher are other things aside from credentials, things like exposed customer pir. For me, that was one of the most shocking discoveries in that we found a major banking institution publishing know your customer information related to customers that contained their name, their address, the characteristics of their mobile device.

A (16:26)

And that was within the code and the neatening of the code PII information.

B (16:34)

Yeah, correct. It was a JSON blob, which is a common format used to store information like that.

A (16:42)

And through all of this you also found a PowerShell script in the midst of all this as well. And how does that fit with these tools?

B (16:54)

That's the exact thing, Christian, that we are purely because these sites are designed to form a JSON data. PowerShell data. So how does PowerShell data end up in a JSON formatting tool? Really great question. I think it's our thought that perhaps the reason this code is in these tools was to share, was to save and generate a link to share with someone out of ease of use. Because these tools wouldn't be used to process this type of code. It has no function for it.

A (17:21)

But PowerShell's a pretty powerful tool and a good way to infiltrate anybody's system, particularly if there are credentials with it. Could you have actually hacked somebody using that PowerShell script?

B (17:35)

Using the PowerShell script in the blob? No. The focus isn't always on gaining credentials for some of the world's highly targeted organizations that are in scope of those tier zero attackers that really will get in by any means necessary. The amount of information in that PowerShell script alone is like gold. Having information on how web applications are deployed inside organizations, having understanding how they're hardened, how their security controls implemented, or even what do common commands or common setup commands look like in these environments is extremely powerful for attackers. So credentials not even being required as an attacker. If I can go equipped and targeting an organization with all of this knowledge, all these kind of prereqs, it's really valuable.

A (18:22)

So you're at your computer doing all this. Did you like do holy shit, what, did you phone somebody? What did you do?

B (18:33)

It's. Yeah, I would say when digging into these new tools that would never be looked at before, you always have the feeling of what and what is going to be here? Uncharted waters. There's probably going to be things here, but I don't think I expected the scale of sensitive information that we found. And we stopped after five years of one of the platforms and one year of the other platforms purely because we were saturated with data. We had more than enough data to responsibly disclose and get underway to start fixing.

A (19:01)

And the one that Just popped out at me was you found Jenkins servers information there as well and one of them and was for mitre which for those who don't know mitre, it's a not for profit organization that's operated, it's federally funded and it does research and development into security in aviation, cybersecurity, healthcare, homeland security. This is a pretty big deal organization and they've got their credentials. They've got credentials or at least secrets and things available in this list.

B (19:40)

Yeah, after digging into it I would say MITRE definitely stood out to us. All of us within security typically recognize those institutions names, but it stood out to us because exactly that MITRE is a recognizable institution. After digging into the data we found exposed, we actually determined it was a university student at a very well known US college who have access to some shared tooling within MITRE that uses their partnership network where they grant access to external contribution.

A (20:09)

We're not accusing them of being loose with this, but any entry into your system, still an entry into your system and whether it comes from a student or whether and hopefully it was a student or someone who makes these types of mistakes, still the enormity of what you found and why that was resident in openly available code just blows me away. You contacted a pile of people, you give a list of them in that you went to organization the uk, Greece, our Canadian center for Cybersecurity, CISA, others and got crickets back. No, no answers, no nothing.

B (20:52)

I'd say it was a really big mix. Ultimately we were dealing with so much information that affected so many organizations, we decided to go straight to the third team that had the overarching responsibility for regions or countries. So reaching out to the likes of NCSE uk, NCSE Norway, NCSE Greece or rather NCSA Greece rather or like you said, the Canadian Centre for Cybersecurity reaching out to these organizations so that they can also help to responsibly disclose these findings to organizations that are within their jurisdiction. Now I think we got some amazing responses from those SERP team. They're incredibly responsive and really good to deal with in terms of helping out, distributing and sending out alerts to affected organizations. We also directly reached out to many organizations that were impacted by a bunch of the high risk items that we found. In some cases many organizations were really responsive, replied to us almost immediately, dealt with it, locked it down, which I think is really amazing to see. However, we reached out to many other organizations that didn't respond or perhaps tried to route us to their bug value programs instead.

A (21:52)

For example, yeah, One place you reached out to said, oh, you've got to actually follow our process for reporting this. You're going, wait a minute, it's your problem, not mine. That would have been, I'm saying it's your problem, not mine. I'm just being a nice guy and being professional about this. Now, in fairness, one of the things that happens in cybersecurity is because every kid and their dog has access to AI now and can go and find what they think is a vulnerability. A lot of these places are snowed under by reports that are mostly generated by AI. AI slop for a lot of them. And they're not highly staffed, they don't have tons of people. So I give them some fairness on that. But when an organization that's in the industry reaches out, I expect you there's. I don't want to call it honor among thieves, but no, there should be a professional response. You say you got that from a lot of organizations and not from some of the companies or groups that you reached out to.

B (22:54)

Yeah, correct. Some of the organizations, incredibly responsive, reaching out immediately, getting the credentials rotated, taken down, things like that, taking steps to mitigate it, which is absolutely what needed to happen. But again, like I said, for many of the organizations we reached out to, we didn't get a response at all or were perhaps again given router to their bugmans programs.

A (23:14)

Now and in case anybody's saying this is all pretty theoretical, you actually did a test to find out how fast it would take people to actually check out the information that was there. So you found. Do you want to just tell us about that?

B (23:30)

Yeah, sure. So as part of this research, we're always asking the question, we're amazing. We've came up with this awesome idea, are we first? And in many cases the answer is no. Of course there are not many novel ideas within security that someone else hasn't already experimented with. And it was our kind of, we wanted to discover had anybody else been looking at these sources of data. Have attackers already been pilfering all the information that people had act anything public? And of course the answer was yes. And so what we did is we published a set of canary credentials, credentials that we control and gets alerted if ever they're used or someone attempts to use them to log into a platform. And so we published them on a few different platforms in various different forms. Something really interesting when we did that was we published them with a 24 hour expiry, meaning that in two years time when perhaps somebody else tries to replicate what we did in this blog and go through the recent links pages iterating to discover as much as possible. They won't discover it, but would actually require someone to be actively looking through the recent links on a regular basis, be gathering them up and then trying those credentials. So we published it with an expiry and then 48 hours after we published those credentials, after they'd expired on the platform, we had someone attempt to use them, which to us indicates somebody else was actively scraping and looking at what are people setting on these platforms and then trying the credentials. Meaning that we aren't first too.

A (24:55)

Yeah. And now. And are you going to get some criticism now for revealing this? That now that obviously this one group or maybe several groups knows about this, but now everybody knows about it. How did you feel about going public with it?

B (25:13)

I think it's a really good ethical question and it's the reason why this research took so long for us to eventually publish. We've been trying to disclose this for well over a year at this point, or approaching a year at this point and reaching out to those necessary certs and organizations. But I think nobody benefits from commoditizing access to these tools if it's a single research organization. And attackers, nobody benefits from you. Especially when we know and have evidence that attackers are already there, they're already looking through these credentials. Nobody benefits. And so by publicizing this research, we're bringing more attention to ensuring that organizations understand the risk associated with these data sources. I think it helps to go that one step further to make sure the attention is there and we start to deal with the potential risks and the exposures that already exist.

A (26:02)

So you basically took almost a year contacting people, making sure that people knew about the risks that you'd found and all of that before you went public. And now because I'm hoping in our audience and our audience is quite large, that there's a lot of people, CISOs or other security people are going to be going down saying wait a minute guys, anybody using these tools? What? There's an old North American show, the Lucy show. Lucy got some splaining to do. But the. Which you may not get in the UK but it North American reference and a really old one at that. Never mind. But somebody's going to call some I hope start going back and saying if you're using these tools and if you're placing secrets into tools that should just never be anything you would post on the Internet no matter how it's done. Right?

B (26:54)

Yeah, correct. I think in many cases they are Victims of convenience. In many cases, we're all rushing to get something done. Or perhaps something has been part of our work for so long or perhaps this is the way it's always been done. In my organization we use these tools, we paste our things here and in many cases I suspect the people accidentally leaking these secrets is not intention. They're doing so just to share it with a colleague. They're just doing it innocently to get their wealth format of kids. They can move on, deal with their.

A (27:23)

Other tasks and sometimes it's a productivity thing and I think we have to balance that. If you're managing a group like a group of developers, or if you're trying to help somebody or communicate with somebody who's managing a group of developers, you have to realize you put a lot of pressure on people to, to cut corners. They'll cut corners and not necessarily cut corners, but they'll do things that make them things faster. One of we had one, we did a story the other day and it was just so obvious that there was an auto run in something and somebody was just so much easier if you use the auto run. And of course that meant that every time it clicked once somebody had managed to infiltrate it just basically spread malware and you didn't even see it happening because who pays attention to that sort of thing. This is something I think we have to try to cope with and still be productive. What are the best practices for this.

B (28:27)

Right now it's a really broad answer, I think depending on the organization. Of course we saw some of the world's largest organizations all the way down to the smallest small medium businesses, even solo developers accidentally leaking credentials and secrets and say the recommendations differ massively depending on maturity of the actual audience. But I think there's a few pieces to either doing these things internally, having best practice guides on how should you use these tools, understanding having kind of the education of what to do, what not to do. I think in many cases developers are well aware of the risks associated with some of the popular tools like I mentioned at the headline makers. But in many cases, perhaps using these other tools without thinking those same risks apply. And don't get me wrong this, these are two tools in the bucket of many. I am more than certain we will continue to see accidental disposal of secrets on other platforms on other tools because that's the reality of them, I think.

A (29:23)

But I do believe somebody listening to this and I don't haven't done, I haven't run development in decades directly I've been an executive for Most of that time. But I would be sitting down with my team today and saying, wait a minute, how much of this do we have happening here? Because it really, these, the concept of these supply chain, these supply chain vulnerabilities in GitHub and all of these things and the ability to ripple those through the entire ecosystem. We all talk about it every single day. That's why I still look at this. I hope I'm not going on too much about this. I still look at this and say how the hell did this happen? And it's an aggregate behavior for many people, but it's still a big vulnerability. Is there, Are we, if somebody was going to critique this, are we being too, am I being too big about this? Is this too alarmist? Is, are we blowing this out of proportion? I think that's a question people would ask. Are we.

B (30:27)

I think it's a reasonable question. I think based on the data that we captured, based on the fact that we had the ability to potentially compromise some of the world's largest organizations, well known names and saw just the amount of secrets, the amount of credentials, the amount of sensitive information that absolutely shouldn't be public within these tools, I think it is an appropriate response to say something is going wrong here. How do we prevent this from happening and what's the root cause? Ultimately a lot of these sites are clearly labeled and say don't upload anything sensitive into these tools. And yet we continue to. I think it's clear that even the largest, most sophisticated organizations can absolutely be victims of convenience in these circumstances.

A (31:08)

And the people we hire and it's not just our own staff. You've got, you've listed a consulting organization in there. I don't want to know which one. I really don't. Just in case it was one of the one that I'm in the alumni for SSL certificate, principal names, key tab credentials, internal passwords, external and internal host names, paths to keys now and they said the good news is they responded immediately when you got a hold of them. But these people have. And a decent sized consulting firm has access to a number of companies and a responsibility to not do that.

B (31:52)

Yeah, I think that's a fair critique. I think ultimately this will always happen regardless of the type of organization you are the organizations you use. I think the accidental exposure of information is something we will always deal with and we need to have the processes in place to deal with, to respond, to ensure that actually these things happen. If sensitive information is accidentally leaked, we have the appropriate controls to mitigate the impact of that? Perhaps we are securely storing our credentials and passwords in vaults. Perhaps we are provisioning just in time credentials so that when they're leak there's no impact. Perhaps we are employing all of these techniques we talk about regularly to ensure if and when this does happen, the impact is minor.

A (32:38)

Wow. Yeah, an interesting thing. So what's next for you? This has obviously occupied a year of your life or and I suspect a fair amount of time, a little bit of frustration now and again. At least it comes out in the text.

B (32:52)

Alex. I would say so. Not only wanting to publish this and bring attention to it, but the level of the things involved in getting it published and of course stretched for quite a while.

A (33:03)

Yeah, yeah. So what's next? What are you working on now?

B (33:08)

It's a good question. We're always working on loads of interesting things. Me personally, I always love discovering new ways to find novel assets that belong to organizations, to find things that are exposed externally that perhaps they didn't know they had. That's my fascination. A little bit of a hoarding mentality perhaps. And so definitely have research coming April soon. We're always publishing research with myself included and all the other researchers here on our watch our Labs blog platform and say we're regularly rapidly responding and dealing with the making industry vulnerabilities that you're participating in the world in which we dive through in a blog style format. How these vulnerabilities came to light and the kind of technical nitty gritty details behind them.

A (33:48)

And I get my podcaster license revoked if I don't ask somebody about AI. Are you doing any work with AI?

B (33:55)

It's a really good question. I think we are definitely using it to power certain capabilities. It's definitely not something that's widely integrated across every single workflow. I think it has its use cases. Absolutely. I also think there are many pitfalls. I think we will continue to see it grow. Maybe the kind of the coming years, the months into these tools, I would.

A (34:15)

Have thought for the type of analysis that you're doing with this massive amount of data would actually have been an interesting thing to run this through again. However, making it more public, possibly through AI systems aren't necessarily that secure, at least in terms of the information that you post in them.

B (34:33)

That's always a worry, right? At least in my paranoid mind, at least especially when dealing with research like this. Submitting content to public tools that inadvertently ends up on the Internet is always something I think anyway. And so perhaps our.

A (34:47)

Don'T want this in the Training data. And again, for those who are listening, this is a significant size of files. And yeah, having this in some AI training data, probably not the best thing to do. Well, this has been great. Jake, thank you so much for dropping in so quickly on this. I wish you the best on this. People want to see this. It's Watchtower spelt without the E. Oh W a T C H T O W r. Go talk to your marketing people, tell them not to do that stuff because no E and you can find it. And there'll be a link on the show notes so that you can pick it up. Any last advice or comments for our audience?

B (35:31)

No, thank you for having me. First of all, ultimately I feel that we're only going to continue to see the risks of accidental data exposure continue to increase. I'm certain perhaps in the new year we'll continue to see a new headline made by similar tools and it's always going to have to. You need to make sure you have the necessary things in place to deal with it, respond to it and detect it.

A (35:51)

Yeah. And for all of you who are watching this and listening to us on a Saturday, make your list for Monday morning. Let's get in there and find out what tools we have and what ability they have to store secrets and information we don't want others to get a hold of. Thanks a lot. That's been our show and I'll be thrilled to have other opinions, things that you want to say about this. Maybe you've got other information and I'll be pleased to make that available to our audience as well. And once again, if you have stories to tell or issues that you want to raise in terms of security, drop me a note. You can find me@technewsday.com just use the Contact Us page and try not to have a tinfoil hat on when you do it. No, I'm just kidding. I sort through. I will try to talk to everybody. I've had some really incredible experiences talking to people in the field and I always love to do it. And if the risk is that I talk to a couple of crazies along the way, so be it. But you make up your own mind. I think Jake's a pretty straight ahead guy and I was listening to what he had to say. Love to hear from you. We'd like to thank Meter for their support in bringing you this podcast. Meter delivers full stack networking infrastructure, wired, wireless and cellular to leading enterprises. Working with their partners, Meter designs, deploys and manages everything required to get performant, reliable and secure. Connectivity in a space. They design the hardware, the firmware, build the software, manage deployments, and run support. It's a single integrated solution that scales from branch offices to warehouses and large campuses to data centers. Book a demo@meter.com CST that's M E T E R.com CST I've been your host, Jim Love. Thanks for listening.