Leaking your AWS API keys, on purpose? [Research Saturday]

Summary

CyberWire Daily - Episode Summary: "Leaking your AWS API keys, on purpose? [Research Saturday]"

Release Date: November 30, 2024
Host: Dave Buettner, N2K Networks

---

Introduction to Research Saturday

In this episode of CyberWire Daily's "Research Saturday," host Dave Buettner welcomes Noah Pack, an intern with the SANS Internet Storm Center. The discussion centers around Noah’s research titled "What Happens When You Accidentally Leak Your AWS API Keys."

---

Noah Pack’s Experience with Leaking Credentials

Noah begins by sharing his personal experience during the onset of the COVID-19 pandemic. With conferences and networking events canceled, he decided to create a Python script to email companies requesting free conference swag.

Noah Pack [02:38]:
"I created a script that would email companies and ask for free conference swag... I posted this code on GitHub shortly thereafter, receiving multiple login requests to the email address I'd created for the script to use."

He admits that, as a freshman in computer science, he had not yet learned secure programming practices, leading him to hard code his email credentials directly into the script. This oversight resulted in unintended exposure when the code was made public on GitHub.

Noah Pack [04:58]:
"There had been no ill consequence. But if my project had been bigger and I was using AWS or another cloud provider and hard coded credentials, that could have dire financial consequences."

---

Understanding Canary Tokens

Dave prompts Noah to explain canary tokens for listeners unfamiliar with the concept.

Noah Pack [06:12]:
"A canary token is sort of like a honeypot... Canary tokens can be things like an Excel document, a QR code, or AWS API keys. When a threat actor opens the document, scans the QR code or uses that API key, it sends an email alert to whoever made the canary token."

Canary tokens serve as deceptive traps to alert owners when sensitive information is accessed maliciously.

---

Research Methodology and Findings

Noah details his research setup, where he embedded AWS API key canary tokens into a modest e-commerce website and later on GitHub.

Noah Pack [07:24]:
"I added some AWS API key canary tokens to about a moderately but small e-commerce website... I also added some AWS API key canary tokens to GitHub."

The initial embedding on the website resulted in slow detection, likely due to the site's lower visibility. In contrast, posting the tokens on GitHub led to an immediate and overwhelming response.

Noah Pack [10:31]:
"The requests just flooded in. It was much different to when I embedded them in the website. I ended up having to turn off the alerts just to preserve my email inbox."

AWS actively scans GitHub for exposed API keys through their secret scanning feature, which promptly tests and revokes compromised keys. Additionally, GitGuardian's service detected the leaked keys, verifying them multiple times within minutes.

Noah Pack [11:42]:
"AWS will test those keys themselves... AWS tested the key, I got a ton of requests from a company called GitGuardian."

---

Impact and Response to Key Leaks

Noah explains the deluge of alerts he received after making the GitHub repository public, including automated scans and threat actor attempts. This rapid response underscores the effectiveness of integrated security measures in detecting and mitigating exposed credentials.

Noah Pack [14:47]:
"This key got a ton of alerts right away... almost all of them were from GitGuardian. There was also the request from AWS and a couple from IP addresses that had been seen doing similar things and scanning the Internet."

To mitigate the issue, Noah promptly removed the GitHub repository and emphasized the necessity of rotating exposed keys to prevent unauthorized access.

Noah Pack [15:15]:
"The best practice is definitely to rotate them to remove all permissions from those keys and create new keys with the permissions that your code needs."

---

Lessons Learned and Best Practices

Noah distills the key lessons from his research:

1. The Severity of Leaked Credentials:
   According to Verizon's 2023 Data Breach Investigation Report, 61% of data breaches stem from leaked credentials.

2. Challenges in Prevention:

   • Reused Credentials: Users often reuse usernames and passwords across multiple platforms.
   • Social Engineering: Even experienced developers can fall victim to sophisticated social engineering attacks.
   • Accidental Exposure: Mistakes like hard coding credentials remain common.

3. Mitigation Strategies:

   • Rotate and Revoke Keys: Immediately rotate leaked credentials to prevent unauthorized access.
   • Implement Canary Tokens: Use canary tokens to detect unauthorized usage of credentials.
   • Educate Developers: Promote secure coding practices to prevent accidental leaks.
   • Monitor and Detect: Utilize tools like GitGuardian and cloud provider logs to identify and respond to leaks swiftly.

Noah Pack [17:37]:
"Leaking your AWS API keys or any credentials is an extremely big deal... [and] if you publish your AWS API keys on GitHub, they will be used... teaching secure coding practices is also probably the best and easiest way to prevent this."

Noah also shares his favorite threat hunting technique—establishing a baseline of normal activity to identify anomalies in network traffic and logs.

Noah Pack [22:10]:
"Take a baseline of something. So on a network I would take a packet capture and look at all of the traffic for the network, slowly eliminating things that I know aren't bad."

---

Conclusion and Final Advice

Noah emphasizes that credential leaks, while seemingly straightforward, pose significant risks and are prevalent across organizations of all sizes. He urges businesses to prioritize credential management and incident response preparedness to mitigate potential breaches effectively.

Noah Pack [19:25]:
"It happens all the time... I've seen horror stories from small businesses that had their AWS account hacked and the attackers racked up bills in excess of $300,000 before the developers could figure out how to rotate those keys and mitigate the problem."

Dave concludes the discussion by highlighting the critical takeaway: even basic security measures, if neglected, can lead to substantial vulnerabilities.

---

Key Takeaways

• Credential Exposure is Critical: Leaked API keys can lead to severe financial and security repercussions.
• Automated Detection is Essential: Tools like AWS’s secret scanning and GitGuardian's services are vital in identifying and mitigating leaked credentials swiftly.
• Best Practices Save Resources: Implementing secure coding practices, rotating keys promptly, and educating development teams are fundamental steps in preventing credential leaks.
• Continuous Monitoring: Establishing baselines and continuously monitoring network and access logs help in early detection of suspicious activities.

---

For more in-depth insights and ongoing discussions on cybersecurity threats and best practices, listeners are encouraged to tune into future episodes of CyberWire Daily.

Transcript

[Dave Buettner]

You're listening to the Cyberwire network, powered by N2K. Hey everybody, Dave here. I want to talk about our sponsor, LegalZoom. You know I started my first business back in the early 90s and oh what I would have done to have been able to have the services of an organization like LegalZoom back then. Just getting all of those business ducks in a row. All of that technical stuff, the legal stuff, the registrations of the business, the taxes, all of those things that you need to go through when you're starting a business, the hard stuff, the stuff that sucks up your time when you just want to get that business launched and out there. Well, LegalZoom has everything you need to launch, run and protect your business all in one place. And they save you from wasting hours making sense of all that legal stuff. Launch, run and protect your business. To make it Official [e-mail] you can use promo code CYBER10 to get 10% off any LegalZoom business information product, excluding subscriptions and renewals that expires at the end of this year. Get everything you need from set up to [e-mail] and use promo code CYBER10. That's legalzoom.com and promo code CYBER10. Legalzoom provides access to independent attorneys and self service tools. Legalzoom is not a law firm and does not provide legal advice except where authorized through its subsidiary law firm, LZ Legal Services llc. Hello everyone and welcome to the Cyberwires Research Saturday. I'm Dave Buettner and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems, and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

[Noah Pack]

It was the beginning of the COVID 19 pandemic. Every conference and career networking event had been canceled that I could find and I created a script that would email companies and ask for free conference swag.

[Dave Buettner]

That's Noah Pack, an intern with the SANS Internet Storm Center. The research we're discussing today is titled what happens when you accidentally leak your AWS API keys.

[Noah Pack]

When I started my time in university I was doing an Introduction to Computer Science class and towards the end of that class the instructor encouraged all the students to pursue a personal project, something they could post publicly on GitHub or learn from. And I saw a video online of a student at a different university who created a Python script to email universities and ask for free swag. So things like a T shirt or a mug. And he got a great response. A lot of admissions Departments sent him things to encourage him to apply there for grad school. I wanted to take that same idea and adapt it a bit. It was the beginning of the COVID 19 pandemic. Every conference and career networking event had been canceled that I could find. And I created a script that would email companies and ask for free conference swag. So I wrote this up in Python. I found a list of 10 companies that I was fond of and I hoped would respond to me with a T shirt or keychain of some kind. I added those company names to my script and it worked flawlessly. I sent it out. I checked the email sent folder and saw all 10 messages. But to celebrate my achievement, I posted this code on GitHub shortly thereafter, receiving multiple login requests to the email address I'd created for the script to use. That was because I hard coded, which means that I put in plain text inside of the code the username and password for that email account for my script to use.

[Dave Buettner]

Noah. Noah. Noah. Dear, sweet Noah.

[Noah Pack]

And I was just a freshman in computer science, so I hadn't learned safe programming practices yet, but I certainly learned from this situation. There had been no ill consequence. But if my project had been bigger and I was using AWS or another cloud provider and hard coded credentials, that could have dire financial consequences.

[Dave Buettner]

Yeah, so the lesson here, I suppose, is that moments after you put this hard coded email address and password up on GitHub, I guess automation from other people had had searched that out and just started hammering that email address, that's exactly what happened.

[Noah Pack]

It happened within minutes. And I saw the same thing in my research. When I published Canary tokens on GitHub for AWS API keys, they were snatched up and used immediately by both threat actors and security companies that were monitoring for them.

[Dave Buettner]

Well, let's dig into the research that you did here. I mean, that this does involve canary tokens for folks who may not be familiar with that. How do you describe a canary token?

[Noah Pack]

Yeah, so a canary token is sort of like a honeypot. At the Internet Storm center, we use honeypots, most of which are Raspberry PIs that look like an attractive target sitting on the Internet for a threat actor to go after. The honeypots record what commands the threat actor uses and what files are downloaded. Canary tokens are similar, but on a much smaller scale. They work really well to supplement the honey pots that we use. Canary tokens can be things like an Excel document, a QR code, or AWS API keys. When a threat actor opens the document, scans the QR code or uses that API key, it sends an email alert to whoever made the canary token, alerting them and giving them a little bit of information about how it was used. In my use case of AWS API key tokens, it gave me the IP address and user agent that tried to use those credentials.

[Dave Buettner]

Well, let's dig into the research here. I mean, what exactly did you set up?

[Noah Pack]

So to conduct my research, I added some AWS API key canary tokens to about a moderately but small e commerce website that I help maintain. It gets roughly a thousand visitors a day. It's enough that it'll come up at the top of a Google search. But someone who's not looking for the things that they're selling probably won't find it. So I didn't really expect this to be found very quickly, and it wasn't. So it took a while before someone picked up those keys and tested them. They could have been picked up much earlier than they were actually tested. But when they were tested, the user agent string was pretty interesting. The person that was testing them was using the Boto3 library and they were using Python on Windows subsystem for Linux and their IP address came from ProtonVPN. So because of the anonymity of a VPN service, it's hard to tie this to any other attacks or to figure out who is actually behind testing this could be anyone from a threat actor who's looking to abuse this credential to a security researcher that's just scanning websites.

[Dave Buettner]

So just so I'm clear here, you had a pre existing website and within this website you embedded the canary token, which to the outside world looked like an AWS API key.

[Noah Pack]

That's exactly right. And I tried to make it look as though a developer might have accidentally left it there.

[Dave Buettner]

And so who do you suppose was going after this? I mean, was this obviously an automated process here?

[Noah Pack]

So I would assume that the key was picked up in an automated process, but that it was manually tested and that if it were, a larger website that receives more traffic, one with a much different threat profile, full automation, or a different threat actor who uses more automation could pick it up much quicker.

[Dave Buettner]

I see. Well, you didn't stop here, you posted your AWS key elsewhere. Take us through the next step of the process here.

[Noah Pack]

That's right. I also added some AWS API key canary tokens to GitHub. Now I created a GitHub repository that I knew any security researcher who lays their eyes on would know that it's a honeypot it's there to catch people. The repository was named Canaries and it had a readme that said something like this is for some research. If you're a bad guy, try these out. If not, please just ignore.

[Dave Buettner]

Wow, okay.

[Noah Pack]

And after making that repository public, the requests just flooded in. It was much different to when I embedded them in the website. I ended up having to turn off the alerts just to preserve my email inbox. But the first one came from aws, the first attempt at using those credentials. And I didn't touch on this in my research, but when you publish AWS API keys on GitHub almost before you can even refresh the page, AWS will test those keys themselves. And it's because GitHub has secret scanning built in where they send anything that they think might be an API key to AWS to test it and AWS will take action. If it's a real API key and not a canary token, they'll send you an email with urgent subject line action required. Your AWS access key is exposed for AWS account and then it will list your account number. But not even seconds after AWS tested the key, I got a ton of requests from a company called gitguardian. Now they have a service that will scan public and private repositories for your secrets. It's a paid service and they tested the keys multiple times within the first few minutes to verify them all from similar IP addresses in Canada.

[Dave Buettner]

We'll be right back. And now a word from our sponsor, KnowBefore. It's all connected, and we're not talking conspiracy theories when it comes to infosec tools, Effective integrations can make or break your security stack. The same should be true for security awareness training. KnowBe4, provider of the world's largest library of security awareness training, provides a way to integrate your existing security stack tools to help you strengthen your organization's security culture. KnowBefore's security coach uses standard APIs to quickly and easily integrate with your existing security products from vendors like Microsoft, CrowdStrike and Cisco 35 vendor integrations and counting Security Coach analyzes your security Stack alerts to identify events related to any risky security behavior from your users. Use this information to set up real time coaching campaigns targeting risky users based on those events from your network, endpoint, identity or web security vendors. Then coach your users at the moment the risky behavior occurs, with contextual security tips delivered via Microsoft Teams, Slack or email. Learn [e-mail] SecurityCoach that's knowbefore.com SecurityCoach and we thank KnowBe4 for sponsoring our show. So they're looking for your business here. They're. They're saying, hey, look what we did. We found this. How. Look how quickly we found this thing. And if you use our service, we'll help you protect against this sort of error.

[Noah Pack]

Not necessarily because they don't reach out to you in any way like AWS did, but they certainly did see it right away and test it out to protect their customers.

[Dave Buettner]

Oh, interesting. Okay.

[Noah Pack]

Yeah, that would be a great marketing strategy, though.

[Dave Buettner]

No, clearly I assumed too much. But that's interesting. So at this point, I mean, you're getting hammered. You said you had to turn off your email because it's just everything's flooding in.

[Noah Pack]

Right. So this key got a ton of alerts right away. Almost all of them were from GitGuardian. There was also the request from AWS and a couple from IP addresses that had been seen doing similar things and scanning the Internet.

[Dave Buettner]

And so what was your response then? I mean, you see the degree to which this has triggered all of this activity. As a researcher, what do you do next?

[Noah Pack]

Yeah, so my next step was to remove the GitHub repository. I had gotten the results that I wanted from my research. I found out that if you publish your AWS API keys on GitHub, they will be used. If you publish them on your website, they will be used. They might take a couple of seconds, it might take a couple of days, but we also don't know the difference between when they're picked up and when they're used. You might be able to rewrite your GitHub repository history and erase those API keys, but someone might still have access to them. They might have downloaded your repository or the source to your website or scanned your website before they use those keys. So the best practice is definitely to rotate them to remove all permissions from those keys and create new keys with the permissions that your code needs.

[Dave Buettner]

Yeah, that was going to be my next question. So, like, once you had removed the information from GitHub, were those keys still being activated? Were there people still trying to hammer away using those credentials?

[Noah Pack]

They were. It took about an hour after removing the repository before my last alert came in. So you could chalk that up to someone having the repository open or downloading it before looking through it. Perhaps their scanner that they're using to find these leaked secrets has a bit of a delay or a bit of a backlog from other code that's being uploaded.

[Dave Buettner]

I wonder if they'll ever get hit again. You know, are there folks out there who will grab this and then say, okay, well, clearly this person realized they had a problem, but we're going to check again in a month just in case.

[Noah Pack]

Oh, I am extremely excited if. Or I will be extremely excited if I see that, because that would be so cool. We know that a lot of threat actors like to lie in wait on networks before they execute their attack, so I'm sure a similar thing is possible here.

[Dave Buettner]

Yeah, well, I mean, I think the lessons here are pretty clear. How do you sum them up for. In terms of the things that you've learned?

[Noah Pack]

Yeah. So leaking your AWS API keys or any credentials is an extremely big deal. According to Verizon's 2023 Data Breach Investigation Report, they said that 61% of data breaches were due to leaked credentials. And while leaking credentials might seem kind of silly, it seems like a fixable problem. I mean, it's the equivalent to leaving keys to a building in the parking lot. But it really is harder to stop than you might think. Users reuse usernames and passwords on sites that are breached. Anyone can fall for a social engineering attack. Even experienced developers can accidentally publish credentials. And all of those reasons are. Those are just three of many reasons that this issue exists and why it's so prevalent, and why entire companies like Truffle Security and GitGuardian exist to solve this problem. I've seen horror stories from small businesses that had their AWS account hacked and the attackers racked up bills in excess of $300,000 before the developers could figure out how to rotate those keys and mitigate the problem. Because they didn't have the incident response experience and they didn't have tools integrated into their code pipeline to find these secrets and stop them from being published.

[Dave Buettner]

It's a really good reminder of what I, you know, I suspect there are folks in our audience who are just nodding along and saying, you know, what a basic, straightforward thing this is. And yet, as you say, despite that, it does happen to so many people.

[Noah Pack]

It happens all the time. There was a cryptocurrency. It was sort of what some people call a meme coin back In, I think 2022 called Shiba Inu Coin and the developers had a code repository on GitHub where they accidentally leaked their AWS credentials. Luckily, some security researchers who are fans of the crypto project found them and unfortunately they had no way to contact the developers. There was no bug bounty program, there was no security Txt on their website. And those researchers noticed that after a few days the AWS API credentials were revoked. They stopped working. Which means that either they did get a hold of someone at Shiba Inu, or the people at Shiba Inu noticed that someone else, maybe a bad actor, was using those credentials.

[Dave Buettner]

Right? What's your advice for folks to help mitigate something like this if it does happen?

[Noah Pack]

The first advice I ever heard on how to mitigate this issue is actually bad advice, and that would be to rewrite your code repository history on GitHub. That's because things like the Wayback Machine exist and you don't know if somebody's downloaded that code with the API keys in it. So the better idea is to rotate those keys. You could also do things like looking at your cloud trail logs in AWS or set up alerts at sans. We like to say that prevention is preferred. Detection is a must, so finding out that those keys were accessed is extremely important. Teaching secure coding practices is also a probably the best and easiest way to prevent this. This includes avoiding the git command, git add, and a wildcard, because that can very easily add sensitive files to your repository. Name the files that contain sensitive information in your gitignore and your NPM ignore files. Those are sort of like the robots Txt of your website, but for git. And then as a threat hunter, one of the techniques that I really like to use is to take a baseline of something. So on a network I would take a packet capture and look at all of the traffic for the network, slowly eliminating things that I know aren't bad. And at the end I'll end up with just the network traffic that could be malicious. And I'll have a bunch of filters that will filter out all the stuff that I know was good. Then I can dig into those things that are bad and do the same exercise again in a month or a week or a quarter and add those same filters and find the new traffic that's bad. That same concept can apply to any log type, including logs from your cloud provider. So look at those cloud shell logs. Understand what is supposed to be running in your AWS account, who is supposed to be running what, and look for services you don't recognize. There are over 200 AWS services at this point, so it's hard to know them all, but you can at least know what ones you use and everything else you can assume is something that you don't, and you can dig into it more.

[Dave Buettner]

Our thanks to Noah Pack from the SANS Internet Storm center for joining us. The research is titled what happens when you accidentally leak your AWS API keys. We'll have a link in the show Notes Imagine this. Your primary identity provider goes down. Whether it's a cloud outage, network issue, or even a cyberattack, suddenly your business grinds to a halt. But what if it didn't have to meet Identity Continuity from Strata, the game changing solution that keeps your business running smoothly no matter what? Whether your cloud IDP crashes or your on prem system faces a hiccup, Identity Continuity seamlessly shifts authentication to a secondary or even tertiary IDP automatically and without disruption. Powered by the Mavericks Identity Orchestration platform, Identity Continuity uses smart health checks to monitor your IDPs availability and instantly activates failover strategies tailored to your needs. When the coast is clear, it's a seamless switchback. No more downtime, no lost revenue, no frustrated customers, just continuous, secure access to your critical application every single time. Protect your business from the high costs of IDP outages. With Identity Continuity from Strata, downtime is a thing of the past. Visit Strata IO Cyberwire to learn how Strata's Identity Continuity can provide seamless, enhanced capabilities to your existing identity fabric. And receive a free set of AirPods Pro. The CyberWire Research Saturday Podcast is a production of N2K networks. N2K strategic workforce intelligence optimizes the value of your biggest investment your people. We make you smarter about your team while making your team smarter. Learn [e-mail] this episode was produced by Liz Stokes. Our mixer is Elliot Peltzman. Our executive producers are Jennifer Iban and Brandon Karp. Our executive editor is Peter Kilpie and I'm Dave Buettner. Thanks for listening. We'll see you back here next time.