Steve Gibson (19:19)
Okay, so Salesforce's name has been dragged back into the news again due to another of their customers. Or I guess API affiliates is probably the best way to explain it. Well, I'll explain all of how that works here in a second. Whose security is not up to snuff. Not Salesforces, but the affiliates. I'm sure Salesforce is very unhappy to have their name dragged back into the news. So almost in response to their like, well, this is not our fault. They posted a very limited stiff acknowledgment. It was under the heading security advisory. Gotta love this unusual activity related to gain site applications. Gainsight being these apps that are causing the trouble. So yeah, unusual activity, right. As in more than 200 customers had their Salesforce stored data compromised. Anyway, their terse posting reads, salesforce has identified unusual activity involving Gainsight published applications connected to Salesforce which are installed and managed directly by customers. Our investigation indicates this activity maybe may have enabled may right we know how to read may have enabled may have for for 200 customers who everyone knows unauthorized access to certain customers Salesforce data through the app's connection. Upon detecting the activity, Salesforce revoked all active access and refresh tokens associated with gain site published applications connected to Salesforce and temporarily removed those applications from the App Exchange. While our investigation continues, there's no indication that this issue resulted from any vulnerability in the Salesforce platform and we'll be coming back to that and you know, giving that a little more attention in a minute. The activity appears to be related to the app's external connection to Salesforce, so right Blaine the Messenger we have notified known affected customers directly and will continue to provide updates. Now all of the available evidence today supports everything technically that Salesforce has said, but TechCrunch provides a much more fulsome account thanks to their reporting of Google's Mandiant Security Group and also Tig, the threat intelligence group. TechCrunch's attention getting headline was Google says hackers stole data from 200 companies following gain site breach, TechCrunch wrote. Google has confirmed that hackers have stolen the Salesforce Store data of more than 200 companies in a large scale supply chain hack. Now that's not the impression that you would get from reading Salesforce's, you know, sort of compulsory admission, TechCrunch said. On Thursday, Salesforce disclosed a breach of quote, certain customers Salesforce data, unquote without naming affected companies. That was stolen via apps published by Gainsight, which provides a customer support platform to other companies. So that's an important datum. Remember that. So this is Gainsight has apps which provide a customer support platform to the users of those apps. In a statement, Austin Larson, the principal threat analyst of Google's Threat Intelligence Group Tig said that the company, quote is aware of more than 200 potentially affected Salesforce instances. After Salesforce announced the breach, the notorious and somewhat nebulous hacking group known as Scattered Lapsus Hunters, which includes the Shiny Hunters gang, claimed responsibility for the hacks in a Telegram channel which TechCrunch has seen the hacking group claimed responsibility for for hacks. Now here's a few of the 200 Atlassian CrowdStrike, DocuSign, F5, GitLab, LinkedIn, Malwarebytes, SonicWall, Thompson, Reuters and Verizon. So not little people we've never heard of before. These are not obscure entities. TechCrunch said Google would not comment on specific victims and as well they shouldn't so TechCrunch did some more digging, writing. CrowdStrike's spokesperson Kevin Banaki told TechCrunch in a statement that the company is, quote, not affected by the Gainsight issue and all customer data remains secure. We'll see how that goes. CrowdStrike confirmed to TechCrunch. Oh, and this is a different story, that it had terminated the employment of a suspicious insider for allegedly passing information to hackers. TechCrunch reached out to all the companies mentioned by scattered lapses hunters. So TechCrunch was going to go, you know, find out who would comment after scattered lapses hunters said, you know, we've got data on the following companies. Verizon's spokesperson Kevin Israel said in a statement that, quote, Verizon is aware of the unsubstantiated claim by the threat actor without providing any evidence for the claim. Malwarebytes spokesperson Ashley Stewart told TechCrunch that the company's security team is aware of the Gainsight and Salesforce issues and actively investigating the matter. The spokesperson for Thomson Reuters said the company is actively investigating. Michael Adams, the chief information security officer DocuSign told TechCrunch in a statement that quote, following a comprehensive log analysis and internal investigation, we have no indication of DocuSign data compromise at this time, unquote. However, Adam said that, quote, out of an abundance of caution, we have taken a number of measures including terminating all Gainsight integrations and containing related data flows. And I'll just note that because this is a breach of Gainsight which is aimed at the Salesforce API, it might well be that there's no logging happening at the breached companies or in the breached companies networks. Which is not to say that their data stored at Salesforce did not leak. So this all feels like, like you know, we've like TechCrunch is asking right on the leading edge of investigations and it's reasonable that this will take some time and also that they might be asking the wrong questions because the companies are. Oh no, our networks are just fine. Well, yes, and that's. It still could be the case that your customer data was leaked from Salesforce. So again, kind of a different question. TechCrunch said hackers with the Shiny Hunters group told TechCrunch in an online chat that they gained access to gain site thanks to their previous hacking campaign that targeted customers of Sales Loft, which provides an AI and chatbot powered marketing platform called Drift. Remember we've talked about that. That's that annoying thing that comes up on the lower right hand side of your screen and says hi, what would you like to talk to me about? Support. And of course it's completely useless, but hey, it's pretends to be support. In that earlier case, the hackers stole Drift authentication tokens from those customers, allowing the hackers to break into their linked Salesforce instances and don't download the contents of their Salesforce store data. So the Shiny Hunters group are saying that they got into in some fashion and gain site did like tokens weren't rotated or refreshed or maybe it was a different, you know, they may have have gotten in and then and then made some lateral moves in order to maintain a grip. That's what they're claiming at the time. Gainsight confirmed it was among the victims of that hacking campaign. So this begins to, you know, feel substantiated. A spokesperson for the Shiny hunters group told TechCrunch, quote Gainsight was a customer of Sales Loft Drift. They were affected and therefore compromised entirely by us. That's Shiny Hunters Salesforce spokesperson Nicole Aranda told TechCrunch that as a matter of policy, Salesforce does not comment on specific customer issues. Gainsight did not respond themselves to TechCrunch's requests for comment. On Thursday, Salesforce said there is no indication that this issue resulted from any vulnerability in the Salesforce platform. Again, as I said, well, very likely true and and TechCrunch said that effectively distanced them from its customers. Data breaches Gainsight has been publishing updates they wrote about the incident on its incident page. On Friday, the company said it is now working with Google's incident response unit Mandiant, to help investigate the breach, that the incident in question originated from the application's external connection, not from any issue or vulnerability within the Salesforce platform and that a forensic analysis is continuing as part of a comprehensive and independent review. Salesforce Salesforce has revoked active access Tokens for gain site connected apps as a precautionary measure while their investigation into unusual activity continues, according to Gainsight's own incident page. So that matches what Salesforce said. And they also said that Salesforce is notifying affected customers whose data was stolen in its Telegram channel. Scattered lapses Hunters said it plans to launch a dedicated website to extort the victims of its latest campaign by next week. This is the group's modus operandi, writes TechCrunch. In October, the hackers also published a similar extortion website after stealing the victim Salesforce data in the Sales Loft incident. And finally, TechCrunch finished their piece writing the scattered lapses Hunters is a collective of English speaking hackers made up of several cyber criminal gangs including Shiny Hunters, Scattered Spider and Lapsus, whose members use social engineering tactics to trick employees into granting the hackers access to their systems or databases. In the last few years these groups have claimed several high profile victims such as MGM Resorts, Coinbase, Doordash and more okay, so I wanted to share this story not only because it's certainly important to those companies who are doubtless scrambling around trying to determine what of their customer data may now be in the hands of extortion happy criminals who are not shy about bragging to the press nor about releasing stolen data they have of that they have managed to acquire. The bigger message I think here is the steadily growing consequences which we keep seeing arising from outsourcing. I'm not suggesting that the benefits do not outweigh the risks, only that risks which remain unseen and unappreciated cannot be hedged against nor planned for When Cloudflare goes down, as we saw two weeks ago, it takes an appreciable portion of the web down with it. The same is true for AWS and Microsoft. From the perspective of the individual customers who've outsourced their needs to those providers, this is an inconvenience for several hours while their critical infrastructure is completely off the Internet. So collectively the amount of pain is huge. But on the other hand, it's very likely that many of those individual providers are positioned behind Cloudflare to obtain the 247 benefits of Cloudflare's bot attack prevention and mitigation. And were it not for Cloudflare on an individual basis, those companies would be periodically blasted off the Internet at the whim of random unknown attackers using today's inexpensive DDoS as a service facilities. So what does this have to do with Salesforce? Same principles are at play here. In the case of Salesforce, the new model is known as BPO Business Process Outsourcing, where significant pieces of a business's operational requirements where there would be, you know, a lot of wheel reinventing without much value to add, are instead of being done in house, developed and done and maintained in house, outsourced to specialist providers. While it makes sense from an operational standpoint to do that, we've just seen more than 200 of Salesforce's individual customers who are all using Gainsight's apps connected to Salesforce's back end API, having their data of an unknown number of their customers exposed through no fault of their own. Recall that long ago there was a story of all of those dental offices that were compromised when the managed service provider that they were all using to outsource a bunch of their dental specific operations, probably dental insurance and dental billing, and their internal networks were all in turn hacked because the msp, the managed service provider they were all using, got breached and the bad guys crawled down the network connections to all of that company's clients. This gainsight Salesforce event is an updated version of that and it's happening at a much greater scale than. Because the services are being, that are being offered have become much more granular and much more generic. You know, sales support desk services. Well, lots of people need that. The general idea is let's not have anything in house that we can subcontract for. It makes companies much more dynamically resizable. And it's far easier to terminate a contract for an outside service that's no longer needed than it is to terminate the employment of a department full of employees with whom multiple birthdays have been celebrated. So the unspoken of cost, the downside of that is that our industry still has very significant operational security problems that show no sign of having been worked out. The fact that Salesforce's reaction to the breach is to invalidate a provider's static access credentials, thus effectively excommunicating them and all of their users from having any access, strongly suggests to me that that today's model of interacting networked applications is still far too crude to withstand the sort of scaling that demand is creating. I'm not close enough to the problems to be able to propose any better solutions, but the way things are being done today feels wrong. I hope those who are in the trenches are thinking about how to make this all work in a more secure and robust fashion. The feeling is that these breaches are still being seen as individual exceptions that hopefully won't repeat. But they are repeating. All the evidence suggests that this is the wrong way to think about them. This feels like, sort of like it's reminiscent of the Internet in the days before the concept of a firewall was introduced, which of course changed the whole landscape. The general concept of widely distributed API linked outsourced services seems to have proven itself. That works. But now the industry needs to figure out how to reduce the blast radius when something evil manages to crawl into the network. What we know is that the more interlinked such complex systems become, the more fragile and vulnerable to malicious exploitation they are. And that's what seems to be happening here. So anyway, and you know, another event of an API user of Salesforce having A breach which in turn allows all of this data stored on behalf of the, of the customers of that API being obtained by bad guys. And you know, we need to prevent that from happening. It's happening over and over and over, which suggests that the model is wrong. The interaction model is like, it's the obvious thing to do, but it's not robust enough. And Leo, I am very excited to share this next piece of news, which was an early Christmas for me. There's nothing, I, I mean, I'm astonished by what Cisco wrote. But let's tell our listeners, okay, why we're here and then we're going to do that.