CyberWire Daily - Episode Summary: "Leaking your AWS API keys, on purpose? [Research Saturday]"

Release Date: November 30, 2024
Host: Dave Buettner, N2K Networks

Introduction to Research Saturday

In this episode of CyberWire Daily's "Research Saturday," host Dave Buettner welcomes Noah Pack, an intern with the SANS Internet Storm Center. The discussion centers around Noah’s research titled "What Happens When You Accidentally Leak Your AWS API Keys."

Noah Pack’s Experience with Leaking Credentials

Noah begins by sharing his personal experience during the onset of the COVID-19 pandemic. With conferences and networking events canceled, he decided to create a Python script to email companies requesting free conference swag.

Noah Pack [02:38]:
"I created a script that would email companies and ask for free conference swag... I posted this code on GitHub shortly thereafter, receiving multiple login requests to the email address I'd created for the script to use."

He admits that, as a freshman in computer science, he had not yet learned secure programming practices, leading him to hard code his email credentials directly into the script. This oversight resulted in unintended exposure when the code was made public on GitHub.

Noah Pack [04:58]:
"There had been no ill consequence. But if my project had been bigger and I was using AWS or another cloud provider and hard coded credentials, that could have dire financial consequences."

Understanding Canary Tokens

Dave prompts Noah to explain canary tokens for listeners unfamiliar with the concept.

Noah Pack [06:12]:
"A canary token is sort of like a honeypot... Canary tokens can be things like an Excel document, a QR code, or AWS API keys. When a threat actor opens the document, scans the QR code or uses that API key, it sends an email alert to whoever made the canary token."

Canary tokens serve as deceptive traps to alert owners when sensitive information is accessed maliciously.

Research Methodology and Findings

Noah details his research setup, where he embedded AWS API key canary tokens into a modest e-commerce website and later on GitHub.

Noah Pack [07:24]:
"I added some AWS API key canary tokens to about a moderately but small e-commerce website... I also added some AWS API key canary tokens to GitHub."

The initial embedding on the website resulted in slow detection, likely due to the site's lower visibility. In contrast, posting the tokens on GitHub led to an immediate and overwhelming response.

Noah Pack [10:31]:
"The requests just flooded in. It was much different to when I embedded them in the website. I ended up having to turn off the alerts just to preserve my email inbox."

AWS actively scans GitHub for exposed API keys through their secret scanning feature, which promptly tests and revokes compromised keys. Additionally, GitGuardian's service detected the leaked keys, verifying them multiple times within minutes.

Noah Pack [11:42]:
"AWS will test those keys themselves... AWS tested the key, I got a ton of requests from a company called GitGuardian."

Impact and Response to Key Leaks

Noah explains the deluge of alerts he received after making the GitHub repository public, including automated scans and threat actor attempts. This rapid response underscores the effectiveness of integrated security measures in detecting and mitigating exposed credentials.

Noah Pack [14:47]:
"This key got a ton of alerts right away... almost all of them were from GitGuardian. There was also the request from AWS and a couple from IP addresses that had been seen doing similar things and scanning the Internet."

To mitigate the issue, Noah promptly removed the GitHub repository and emphasized the necessity of rotating exposed keys to prevent unauthorized access.

Noah Pack [15:15]:
"The best practice is definitely to rotate them to remove all permissions from those keys and create new keys with the permissions that your code needs."

Lessons Learned and Best Practices

Noah distills the key lessons from his research:

1. The Severity of Leaked Credentials:
   According to Verizon's 2023 Data Breach Investigation Report, 61% of data breaches stem from leaked credentials.

2. Challenges in Prevention:

   • Reused Credentials: Users often reuse usernames and passwords across multiple platforms.
   • Social Engineering: Even experienced developers can fall victim to sophisticated social engineering attacks.
   • Accidental Exposure: Mistakes like hard coding credentials remain common.

3. Mitigation Strategies:

   • Rotate and Revoke Keys: Immediately rotate leaked credentials to prevent unauthorized access.
   • Implement Canary Tokens: Use canary tokens to detect unauthorized usage of credentials.
   • Educate Developers: Promote secure coding practices to prevent accidental leaks.
   • Monitor and Detect: Utilize tools like GitGuardian and cloud provider logs to identify and respond to leaks swiftly.

Noah Pack [17:37]:
"Leaking your AWS API keys or any credentials is an extremely big deal... [and] if you publish your AWS API keys on GitHub, they will be used... teaching secure coding practices is also probably the best and easiest way to prevent this."

Noah also shares his favorite threat hunting technique—establishing a baseline of normal activity to identify anomalies in network traffic and logs.

Noah Pack [22:10]:
"Take a baseline of something. So on a network I would take a packet capture and look at all of the traffic for the network, slowly eliminating things that I know aren't bad."

Conclusion and Final Advice

Noah emphasizes that credential leaks, while seemingly straightforward, pose significant risks and are prevalent across organizations of all sizes. He urges businesses to prioritize credential management and incident response preparedness to mitigate potential breaches effectively.

Noah Pack [19:25]:
"It happens all the time... I've seen horror stories from small businesses that had their AWS account hacked and the attackers racked up bills in excess of $300,000 before the developers could figure out how to rotate those keys and mitigate the problem."

Dave concludes the discussion by highlighting the critical takeaway: even basic security measures, if neglected, can lead to substantial vulnerabilities.

Key Takeaways

• Credential Exposure is Critical: Leaked API keys can lead to severe financial and security repercussions.
• Automated Detection is Essential: Tools like AWS’s secret scanning and GitGuardian's services are vital in identifying and mitigating leaked credentials swiftly.
• Best Practices Save Resources: Implementing secure coding practices, rotating keys promptly, and educating development teams are fundamental steps in preventing credential leaks.
• Continuous Monitoring: Establishing baselines and continuously monitoring network and access logs help in early detection of suspicious activities.

For more in-depth insights and ongoing discussions on cybersecurity threats and best practices, listeners are encouraged to tune into future episodes of CyberWire Daily.