Steve Gibson (137:45)
And the nih, the US National Institutes of Health. For those who are not tied into the enterprise world, it may not be intimately familiar with Microsoft SharePoint. Microsoft says that SharePoint enjoys 200 million users. That's not servers, that's people using it. But here's how Wikipedia describes it in two lines. They said SharePoint is a web application by Microsoft that's primarily used for building an intranet and managing and sharing files. Launched in 2001, the Year of the Space Odyssey, it was initially bundled with Windows Server as Windows SharePoint Server, then renamed to Microsoft Office SharePoint Server, and then finally renamed just SharePoint. It could be used on premises or as a Microsoft 365 hosted service, you know, in the cloud. So this news was breaking while we were recording last week's podcast. I don't know why that's happening now, Leo, but like for this is the second time, second week in a row was like, while, you know, like last last week while we were recording the podcast, Cloud strikes Outage was, you know, I mean, cloud flare, sorry, Cloud flare's big DNS outage was happening for an hour. So today's news, while we were, while we're, we were recording last week's podcast. Anyway, enough time has now taken for the story to have taken shape, so I'm going to share first what Wired wrote, since it nicely places the story into context and provides some background. And after that, we'll examine what the security firms who dug into this more deeply found. So Wired said hundreds of organizations around the world suffered data breaches as an array of hackers rushed to exploit a recently discovered vulnerability in older versions of the Microsoft file sharing tool known as SharePoint. The string of breaches adds to an already urgent and complex dynamic. Institutions that are longtime SharePoint users can face increased risk by continuing to use the service, just as Microsoft is winding down support for this platform in favor of newer cloud offerings. In other words, as I've said before, Microsoft is like saying, sorry, no, we're no longer going to support the things you bought from us in the past. Now you're going to have to subscribe to the same thing in the cloud, they wrote. Microsoft said last Tuesday that in addition to other actors, it has seen multiple China Linked hacking groups exploiting the flaw, which is this is Microsoft admittedly acknowledging this, which is specifically present in older versions of SharePoint that are self hosted by organizations. In other words, using SharePoint Server 2016 and 2019 or maybe even older ones. It does not impact the newer cloud based version of SharePoint that Microsoft has been encouraging customers to adopt for many years. Bloomberg first reported on Wednesday that's last Wednesday that one of the victims is the United States National Nuclear Security Administration, which oversees and maintains US's nuclear weapons on premises or self managed SharePoint servers are a popular target for hackers because organizations often set them up such that they are exposed to the open Internet and then forget about them or don't want to allocate budget to replace them. That sound familiar? Oh, even if fixes are available, the owner may neglect to apply them. This is Wired magazine, right? This is not this podcast. We say that all the time, wired said. That's not the case though, with the get this with the bug that sparked this week's wave of attacks. While it relates to a previous SharePoint vulnerability discovered at the PWN to Own hacking competition in Berlin in May, the patch that Microsoft released earlier this month was itself flawed, meaning even organizations that did their security diligence were caught out. Microsoft scrambled this week to release a fix for the fix, or what the company called more robust protections in its security alert. Now I'll just pause to say this really shouldn't surprise us. We've covered in the past how Microsoft's current incarnation of security updates appears to focus upon implementing a quick fix for the symptoms rather than addressing underlying systemic weaknesses. I don't know that's what happened in this instance, but if it quacks like a duck, Wired continued writing A Microsoft spokesperson wrote in an emailed statement, quote, at Microsoft, our commitment anchored in the Secure Future initiative is to meet customers where they are. That means supporting organizations across the full spectrum of cloud adoption, including those managing on premises systems. Wow. Okay. Talk about statement that says nothing. Anyway, Wired continues, Microsoft still supports SharePoint server versions 2016 and 2019 with security updates and other fixes, but both will reach what Microsoft calls end of support on July 14, 2026. Well, okay, this is the 20th. This is the 29th. So that happened SharePoint Server 23rd, 20. I'm sorry, 2026. So next year. What am I saying? This is 2025. So July 14, 2026. So just just short of a one year from now, support for 2016 and 2019, SharePoint servers will end, they wrote. SharePoint Server 2013 and earlier have already reached end of life and receive only the most critical security updates through a paid service called SharePoint Server Subscription Edition. Right. So you could subscribe to as now Microsoft is doing to receive extended support Wired wrote. As a result, all SharePoint server versions are increasingly part of a digital backwater where the convenience of continuing to run the software comes with significant risk and potential exposure for users, particularly when SharePoint servers sit exposed on the Internet. Jake Williams, a longtime incident responder who's vice president of research and development and hunter strategy, said, quote, years ago Microsoft positioned SharePoint as a more secure replacement for old school Windows file sharing tools. So that's why organizations like government agencies, maybe the CIA invested in setting up those servers and now they run at no additional cost compared to Microsoft. 365 subscription in the cloud that requires continuous payment. Okay, this is, this is, you know, this is not me saying this again. This is somebody else. So no surprise, he says. So Microsoft tries to nudge the holdouts by charging for extended support. But if you're exposing a SharePoint server to the Internet, he said, I would emphasize that you also have to budget for incident response because that server will eventually get popped, unquote. Wired says the United States Cyber Security and Infrastructure Security Agency said in a in guidance about the vulnerability Tuesday that, quote, CISA recommends disconnecting public facing versions of SharePoint server that have reached their end of life or end of service. For example, SharePoint Server 2013 and earlier versions are end of life and should be discontinued if still in use. Now the problem is it's working and it's been paid for. So when budgets are tight and when are they not going through all the hassle of switching to a paid Microsoft cloud based service and then needing to continue paying for it can be a difficult sell to upper management. As I've observed here recently, the entire model that's evolved across our industry of selling online software systems that are later found to have critical vulnerabilities and expecting their users to suddenly take proactive responsibility or even be aware that there's a problem that needs their attention is inherently impractical and is badly broken in practice. Wired's author of this article apparently agrees, writing, the ubiquity of Microsoft's Windows operating system around the world has led to other situations in which a long goodbye is the way he put it. A long goodbye has created security issues for holdout users and other organizations or individuals with connections to a vulnerable entity. Microsoft struggled to deal with the long tail of users on extremely popular Windows editions, including Windows XP and Windows 7. And of course I would expect this whole drama to repeat itself with Windows 10 starting soon, wired wrote. But legacy software is a challenge for any software or digital infrastructure provider. Earlier this year, for example, Oracle reportedly notified some customers about a breach after attackers compromised a legacy environment that had been largely retired in 2017, yet people were still using them. The challenge with a service like SharePoint is that it often acts as an ancillary tool without ever being the center of attention, meaning it's just kind of there in the proverbial back closet somewhere, working and forgotten. Bob Huber, chief security officer at the cybersecurity company Tenable, says quote for on premises software like SharePoint, which is deeply integrated into the Microsoft Identity stack, there are multiple points of exposure that need to be continuously monitored in order to know, expose and close critical gaps, unquote. When asked about the alleged breach at the nuclear at the National Nuclear Security Administration, the Department of Energy emphasized that the incident did not impact sensitive data or classified data. A DOE spokesperson told Wired in a statement, quote On Friday, July 18, the exploitation of a Microsoft SharePoint Zero Day vulnerability began affecting the Department of Energy, including the nnsa. The department was minimally impacted due to its widespread use of the Microsoft 365 cloud and very capable cybersecurity systems. So a bunch of them had migrated to the cloud, but not all, he said. A very small number of systems were impacted and NSA is taking the appropriate action to mitigate risk and transition to other offerings as appropriate. So maybe this, you know, this incident has spurred people to move to the cloud. Microsoft Wired Finishes say Microsoft did not immediately return WIRED's request for comment about the process of sun setting SharePoint Server. The company wrote in a blog post on Tuesday that customers should keep supported versions of SharePoint Server updated with the latest patches, although that didn't help in this case, and turn on Microsoft's Anti Malware scan interface as well as Microsoft Defender antivirus. Unfortunately, as we saw, we're going to get some more information on this now. Microsoft fumbled and botched the security patch during May's Pone to Own competition, which we covered at the time since it was the first time Pone to Own had been held in Berlin. Having moved from Toronto, a researcher with with Cyber Security with the cyber security arm of Vittel, a telecom firm run by Vietnam's military, identified a SharePoint bug dumb dubbed Tool shell and demonstrated a way to exploit it. That discovery won the researcher an award of $100,000. But here's where the plot thickens. Exploits discovered by security researchers remain explicitly secret. We only learned that there is a flaw and of its general nature and nothing more. As part of the researchers agreement, they confidentially provide all required information to Trend Micro Zero Day Initiative team, which then in turn forwards that information to the affected software vendor with a 90 day time expiration on that zero day flaw being patched. The publication we all know the Register theorizes that the exploit may have leaked from Microsoft. I don't, I don't buy into it completely, but here's what the Register said they said less than two months later, on July 8th, Microsoft disclosed the two CVEs 49704, which allows unauthorized remote code execution, and 49706, a specific spoofing bug, and released software updates intended to patch the flaw. So so July 8th was Patch Tuesday earlier this month, the Register wrote. But mass exploitation had already started the day before on July 7th. Now that's not true. Some exploitation had started, not the mass exploitation. Dustin Childs quotes the Register head of Threat Awareness at Trend Micro Zero Day initiative said quote 60 days to fix isn't a bad timeline for a bug that stays private and stays under coordinated disclosure. What is bad is that a leak happened which is which may have been true. Again, no proof of it. Patch Tuesday happens the second Tuesday of every month, writes the Register. In July. That was the eighth. But two weeks before then Microsoft provides early access to some security vendors via the Microsoft Active Protections program mapp. These vendors are required to sign a non disclosure agreement about the soon to be disclosed bugs and Microsoft gives them early access to the vulnerability information so that they can provide updated protections to customers faster, Childs said. Childs with Trend Micro said the first MAPP drop occurs at what we call R minus 14, meaning 14 days before release release minus 14. In this case that was June 24th. Then on July 7th, which was one day before the patch Tuesday, he said we started to see attacks again not apparently mass and widespread. July 8 the patches were out and were almost immediately bypassed. Well they were almost immediately bypassed for another reason. ZDI writes the Register along with other security providers poked holes in the initial patches true and determined that the authentication bypass piece was too narrow and attackers could easily bypass this fix. In fact, writes the Register, anyone who received the early MAPP information about the CVEs and software updates would be able to tell that this is an easy way to get past it. I want to make sure I don't forget to mention that once the Patch Tuesday patches came out they were immediately diffed. You know differences made and the patches were reversed in reverse engineered from the diffs not from any MAPP or not necessarily using any MAPP advanced information. So you Know a lot is still unknown. The register finishes saying on July 18, iSecurity first sounded the alarm. So it was on the 18th of July, 10 days later, 10 days after patch Tuesday, that the actual large scale exploitation of a new SharePoint remote code execution vulnerability chain in the wild was seen. And. And one day later, that's when Microsoft warned SharePoint server users that three on prem versions of the product included a zero day flaw. Including a zero day flaw was under attack and that its own failure to completely patch the holes was to blame. Okay, but there's more. Shodan shows that around 8,000 SharePoint servers in use by auditors, banks, health care companies, major industrial firms and US state, federal and international government bodies are in use. 8,000. In other words, it's a mess. The 8,000 figure might be conservative because the Shadow Server foundation, which continuously scans the Internet for potential digital vulnerabilities, put the number at a little more than 9,000, cautioning that that figure is a minimum. That is a minimum of more than 9,000 instances of SharePoint of vulnerable SharePoint on the Internet, meaning that they were able to confirm that 9,000 figure. And there's like, there's likely more. The Shadow Server foundation said most of those affected were in the United States and Germany. The last thing I want to share is some of the very interesting reporting by the researchers at I Security. Those were the first people to report this and to report that basically the attacks went viral. And the last thing you want is it just to hear that a zero day remote code execution vulnerability attack has gone viral. Remember that the first known instance of exploitation occurred on the day before the July 8th patch Tuesday, which was exactly three weeks ago today. As we're recording today's podcast, the Isecurity researchers wrote on the evening of July 18, iSecurity was the first to identify large scale exploitation of a new SharePoint remote code execution vulnerability chain in the wild demonstrated just days before on X, which is, you know, what used to be called Twitter, the X platform. This exploit is being used to Compromise on premise SharePoint servers across the world. The new chain we uncover in this blog was later named and then they updated CVEs. Microsoft had the first round of CVEs. Then when they fixed the fix they gave them CVEs 53770 and 53771 they wrote before this vulnerability was widely known. Last Friday our team scanned over, get this, 23,000 SharePoint servers worldwide in total at that time. That is again before this vulnerability was widely known. Last Friday they discovered more than 400 systems actively compromised during four confirmed waves of attack. So there was so four confirmed waves of attack. They enumerate them the initial attack wave 17th of July at 12:51 UTC coming from IP 96.912-5147. They they think that was a testing wave to verify their exploit. Then there was attack wave one the next day on the 18th of July at 1806 UTC coming from a different IP 107191 58, 76 and that one was widely successful. The next the the the third wave the the second big attack wave the the following day 19 July at 728 UTC originating from IP 104, 238, 159, 149 and then multiple smaller waves on and after 21 July after a public proof of concept exploit script was released on GitHub. So basically the world knew about it at that point. And remember all of those attacks were effective against both unpatched, that is Never before patched and then fully patched post Patch Tuesday SharePoint servers. So there was a 10 day window from July 8th through July 18th when these attacks were effective and even after the 19th before the updated update got out to everybody. So in this instance it was Microsoft's significant fumble of the initial patches that were readily bypassed because they were merely cosmetic symptom covering patches of the sort that we've seen before. There are postings on the net by people saying that they diffed Microsoft's Patch Tuesday patches. That is not members of the MAPP program who received them from two weeks before Patch Tuesday. But they got the the Patch Tuesday updates, found out what the differences were that affected SharePoint server and then reverse engineered the patches and wrote code to sidestep the IM the imperfect fixes that Microsoft had attempted to implement in Patch Tuesday. So basically Microsoft made things much worse by by poorly patching SharePoint server on Patch Tuesday because then everybody else in the world was able to see what they changed, take a close look at it and see that Microsoft had not actually fixed the problem. Note also that Microsoft's updated patches which do now the updated patches do now actively res. You know, actually resolve the problem. Those only cover SharePoint Server 2016 and 2019. SharePoint Server 2010 and 2013 which are on the Internet, remain vulnerable and no patch for those is expected. So they must either be isolated from the public Internet or shut down. Okay, so what exactly. And this is really cool because it's a sort of A here's how like exactly what happened this here. Here's what the I security guys saw that led into this whole thing, they wrote early in the evening, our 24. 7 detection team received an alert from one of our CrowdStrike Falcon EDR deployments at a specific customer. The alert flagged a suspicious process chain on a legacy SharePoint on prem server tied to a recently uploaded malicious aspx file. At first glance, it looked familiar. A classic web shell obfuscated code in a custom path designed to allow remote command execution via HTTP. We've seen many of these before. What made this one stand out, however, is how it got there. Our first hypothesis was mundane but plausible. A brute force or credential stuffing attack on a federated active directory identity followed by an authenticated upload or remote code attempt using valid credentials. The affected SharePoint server was exposed to the Internet and tied to Azure Active Directory using a hybrid ADFS that stack when misconfigured or outdated, can be a dangerous combination. It all seemed to confirm the theory. Credentials compromised, shell dropped, persistence achieved but examining the IIS logs the web server Examining the IIS logs more closely, we noticed that the referrer was set to slash underscore layouts forward. You know, forward/signout aspx. That's odd, they wrote. How can that be an authenticated request just after the user has logged out? Something didn't add up. We found no successful authentications in ADFS logs, or the logging was at least insufficient. Malicious IIS logs did not contain a value in the username column. A post request to the underscore layouts/15/toolpane aspx seemed rather specific. Referrer set to underscore layouts under or forward/signout aspx cannot be authenticated right. We began to develop a feeling that credentials were never used. So how could the attacker write files to the server without ever authenticating at all? That's when we realized we were no longer dealing with a simple credential based intrusion. This wasn't a brute force or phishing scenario. This was zero day territory. After some digging, we learned that three days earlier the offensive security team from Code White GmbH demonstrated they could reproduce an unauthenticated RCE exploit chain in SharePoint, a combination of two bugs originally presented at Pwn to Own Berlin earlier this year. In May, those bugs were still present in the patched SharePoint server they dubbed the chain tool shell. What we discovered on the 18th was not a credential issue. We had stumbled upon a weaponized pwn to own exploit already being used in the wild. When our team began reviewing the impacted systems, we expected to find the usual suspects standard web shells designed for command execution, file uploads, or lateral movement. Instead, what we discovered was more subtle and arguably more dangerous a stealthy spinstall 0. Aspx file whose sole purpose was to extract and leak cryptographic secrets from the SharePoint server using a simple get request. This wasn't your typical web shell. There were no interactive commands, reverse shells, or command and control logic. Instead, the Page invoked internal. NET methods to read the SharePoint server's machine key configuration, including its validation key. These keys are essential for generating valid view state payloads, and gaining access to them effectively turns any authenticated SharePoint request into a remote code execution opportunity. Then it all clicked once the machine key configuration, including the validation key had been obtained. Future payloads can embed any malicious commands and would be accepted by the server as trusted input, completing the RCE chain without requiring any credentialing. This mirrors the earlier SharePoint design weakness exploited four years ago in 2021, but it's now been packaged into a modern zero day chain with automatic shell drop and full persistence with zero authentication. More than 24 hours after we published our initial findings and reached out to affected vendors, including Microsoft, the Microsoft Security Response center issued an official advisory and assigned vulnerability identifiers on their page. Microsoft confirmed active exploitation in the wild and acknowledged the severity of the issue. They make one final crucial point. Due to the fact that this is a machine key exfiltration attack, they said. The attack we've observed specifically targets the exfiltration of of SharePoint Server ASP dotnet machine keys. These keys can be used to facilitate further attacks, even at a later date. It is critical that affected servers rotate SharePoint Server ASP net machine keys and restart IIS on all SharePoint servers. Patching alone is not enough if you are not targeted or you are unsure. We also advise teams to rotate their machine keys just to be sure it has no system impact, only that IIS is offline for some seconds while restarting services. So we don't know how many systems, enterprises, organizations and networks have been compromised as a result of Microsoft's botched patches for the original pwn to own zero day. But that number lies somewhere between the 400 that have been confirmed and the 9,000 that were known to be vulnerable by the Shadow Server Foundation. And the attackers were aggressive and automated. This was, you know, this is a, this is a great deal of damage and the ransomware demands have Already begun as an industry. We need to do better, we need to change the model. And of course this has been an important high profile incident. That individual. So much so that individual reports have now been published by Broadcom's semantic CISA, Cisco, Talos Census Checkpoint, CrowdStrike, iSecurity, Log Point, Microsoft Orange, Palo Alto Networks, Qualus Sentinel 1 Tenable, Trend Micro and Veronis another.
