SN 1035: Cloudflare's 1.1.1.1 Outage - Bypassing Passkey Protections

Summary

Security Now (Audio), Episode 1035

Title: Cloudflare's 1.1.1.1 Outage – Bypassing Passkey Protections
Date: July 23, 2025
Hosts: Steve Gibson & Leo Laporte

Episode Overview

In this insightful episode, Steve Gibson and Leo Laporte unpack a dramatic global DNS outage caused by a misconfiguration at Cloudflare, delving into the technical nuances and the resulting worldwide impact. The show also covers the discovery of a viable attack for bypassing the much-vaunted passkey protections, explores the growing governmental clampdown on internet freedoms (with a focus on Russia and China), and examines the persistent industry challenges around age verification for internet users. Additionally, the episode highlights cybersecurity trends, notable ransomware attacks, changes in corporate security policies, and features Steve’s ongoing work with DNS benchmarking tools.

---

Major Discussion Topics & Insights

1. Cloudflare’s Quad One (1.1.1.1) DNS Global Outage

Timestamp: [04:14], Detailed Discussion: [128:18], [151:55]

• On July 14, 2025, Cloudflare's flagship DNS resolver “1.1.1.1” (“Quad One”) went offline globally for about an hour due to an internal misconfiguration associated with BGP (Border Gateway Protocol). Both IPv4 and IPv6 addresses were impacted, making DNS resolution for millions of users unavailable.
• The error stemmed from inadvertently including the Quad One service prefixes in a topology intended for a non-production DLS service. This latent configuration, introduced in June but only triggered in July, led to Cloudflare withdrawing critical BGP announcements worldwide.
• Steve highlighted the crucial best practice: always configure at least two DNS resolvers from different providers to guard against outages like this.
• Quote:

  "This outage occurred because of a misconfiguration of legacy systems used to maintain the infrastructure that advertises Cloudflare's IP addresses to the Internet.... The root cause was an internal configuration error and not the result of an attack or a BGP hijack." – Steve Gibson reading Cloudflare’s report [132:50]

• The use of Anycast IP addresses by Cloudflare means DNS traffic is intelligently routed to the closest data center, usually providing low latency and high reliability—unless all global announcements vanish as they did here.
• The event graphically showed the internet’s reliance on DNS and how a single point of failure can have massive repercussions.
• Quote:

  "For many users, not being able to resolve names using 1111 resolver meant that basically all Internet services were unavailable." – Cloudflare, quoted by Steve [129:58]

2. Active Attacks Successfully Bypassing All Passkey Protections

Timestamp: [17:31]

• Researchers at Expel Security discovered attackers leveraging "adversary-in-the-middle" (AitM) downgrade attacks to bypass FIDO2/passkey (passwordless) authentication during phishing.
• The method involves tricking victims into entering credentials at a phishing site, which then requests a cross-device passkey sign-in, serving the real authentication QR via the fake site. When the user scans this QR code (thinking it’s legitimate), the attackers gain access.
• Key vulnerability: Cross-device passkey flows lack the channel binding that on-device authentication offers.
• Quote:

  "Bad actors have figured out how to downgrade FIDO key authentication when compromising accounts... This technique is being leveraged in phishing attacks, meaning it's happening in the wild that passkeys are being bypassed." – Steve Gibson [17:39]

• Steve contrasted this with his own SQRL technology, noting it was designed to avoid this AitM weakness during same-device authentication.

3. Other Security News & Trends

Ransomware Attacks Remain Pervasive

Timestamp: [33:44]

• Recent ransomware hits included South Korea’s Seoul Guarantee Insurance and the US grocery distributor United Natural Foods, collectively resulting in operational chaos and hundreds of millions in losses.
• A notable legal response: Qantas obtained an injunction against publishing their stolen data, which Steve wryly noted is "unlikely to concern the actual criminals."

Cloudflare Compromises on Net Neutrality – UK Domain Blocking

Timestamp: [39:42]

• For the first time, Cloudflare is actively blocking access to “pirate” sites in the UK, complying with legal orders and returning HTTP 451 ("unavailable for legal reasons") error messages.
• The show discusses the complexities of geofencing, transparency, legal compliance, and the relative ineffectiveness (and new effective uses) of DNS-level blocking.
• Quote:

  "Cloudflare appears to have changed their long standing policy of enforcing total net neutrality... they're now geo-blocking sites for UK users to comply with legal orders." – Steve Gibson [39:42]

Tightening of Internet Controls: Russia & China

Timestamp (Russia): [79:50]

• Russia criminalizes searching for as well as sharing “extremist” content, with penalties and potential for escalation to criminal prosecutions, illustrating increasing global governmental intervention in internet freedoms.
• Quote:

  "A new Russian law has criminalized online searches for controversial content... the new amendments follow remarks by high-ranking officials that censorship is justified in wartime." – Steve Gibson [79:50] Timestamp (China): [92:11]

• Chinese border and police authorities are extracting data from Android phones using new forensics tools; travelers are advised to use burner devices and establish plausible histories.
• Leo notes, "…the USA is just as bad" regarding digital device checks at the border.

The (Still) Intractable Problem of Age Verification

Timestamp: [59:50], [110:02]

• Platforms continue facing mounting pressure to implement robust user age verification (e.g., Roblox, Steam, and payment processor-imposed bans on content).
• Robust technical solutions that preserve privacy while resisting spoofing (fake IDs) are lacking; Steve sketches potential protocols involving biometrics and cryptographic zero-knowledge proofs.
• Quote:

  "The need to verify the age of internet users is not off in the future… we need standards, we need technology, and we need all that yesterday–because the need for age verification is today." – Steve Gibson [61:51]

4. Listener Feedback

Timestamp: [98:54], [109:59]

• Users report successful use of SpinRite to recover Kindles and TiVo devices.
• Discussion about the nature of AI and simulated intelligence—Steve echoes Edsger Dijkstra’s sentiment:
• Quote:

  "The question of whether computers can think is just as relevant and just as meaningful as the question of whether submarines can swim." – Edsger Dijkstra, recounted by Steve [109:21]

5. Steve’s DNS Benchmark Development

Timestamp: [128:40], [166:25]

• Steve provides an update on the next-generation DNS Benchmark tool. It now includes domain filter checking, supports all DNS protocols, and identifies the fastest provider for any user location. He notes Cloudflare’s dominant speed and reliability, largely owing to their strategic use of Anycast.

---

Notable Quotes & Memorable Moments

• On Anycast and Cloudflare’s Outage:

  “It is so crucial to the operation of all of the services that we now just take for granted on the Internet.” – Steve Gibson [166:19]

• On Cross-Device Passkey Phishing:

  “A determined attacker in the middle that’s able to engineer a spoofed phishing attack and convince a user to enter their valid username and password… can still get themselves authenticated even with passkey protected authentication.” – Steve Gibson [30:04]

• On the Changing Internet Regulatory Environment:

  “The world is starting to wake up to the Internet and the age of the people using it is suddenly a big deal. So we need protocols… and I don’t see any sign of this happening.” – Steve Gibson [60:52]

---

Key Timestamps for Important Segments

| Timestamp | Topic Description | |------------------|--------------------------------------------------| | [04:14] | Episode agenda and Quad One DNS outage preview | | [17:31] | Passkey bypass attack: method and implications | | [33:44] | Ransomware attack round-up, Qantas legal action | | [39:42] | Cloudflare net neutrality compromise, UK blocks | | [59:50] | Growing pressure and complexity of age verification | | [79:50] | Russia criminalizing online searches | | [92:11] | China’s phone forensics, travel cyber hygiene | | [98:54] | SpinRite success stories, simulated intelligence | | [109:21] | Dijkstra on AI and intelligence | | [128:18] | In-depth analysis: Cloudflare’s 1.1.1.1 outage | | [151:55] | Cloudflare’s outage timeline, internal causes | | [166:25] | DNS Benchmark tool update |

---

Additional Technical Explanations

• Anycast DNS (Cloudflare): Ensures global low-latency query resolution by routing requests to the nearest geographic data center.
• BGP (Border Gateway Protocol): Vital for internet routing; a misannouncement or withdrawal can “disappear” global services.
• FIDO2/Passkey Bypass: AitM attacks can exploit the fallback “cross-device” authentication flows that lack secure key-binding between authentication factors.

---

Final Thoughts

Steve and Leo close with reflections on the unprecedented dependence on DNS reliability, the need for urgent innovation in age verification protocols, and the rising global tide of state-level internet controls. Steve promises further updates on his DNS Benchmark tool and continues to advocate for privacy-first, user-centric security designs in a rapidly shifting digital landscape.

---

For more resources and this episode’s transcript:

• GRC.com Security Now Show Notes & Benchmarks
• Cloudflare Radar for 1.1.1.1 (shortcut: GRC.SC/1035)

Next Episode Preview: Listener feedback, new security tools, and continued coverage of emergent cyber threats.

Transcript

[Leo Laporte]

It's time for Security Now. Steve Gibson is here with news of a passkey bypass. Yikes. We'll also talk about problems Cloudflare had with its DNS provider and explain what happened. Russia clamps down really clamps down on Internet usage. And some advice if you're planning to travel to China or frankly, anywhere. Stay tuned. Security now is next. Today's show is brought to you by Progressive Insurance. Do you ever think about switching insurance companies to see if you could save some cash? Progressive makes it easy to see if you could save when you bundle your home and auto policies. Try [e-mail] Progressive Casualty Insurance Company and affiliates. Potential savings will vary. Not available in all states. Let's map out this week's amazing destinations and travel tips.

[Will]

Honestly, Will, I didn't plan any trips, but I did switch to T Mobile with their new Family Freedom offer.

[Leo Laporte]

That's not the itinerary we're following.

[Will]

Well, I'm departing from ATT and embarking on a new journey with T Mobile. They paid off my family's four phones up to 3, $200 and gave us four new phones on the house.

[Leo Laporte]

Bon voyage.

[T-Mobile Announcer]

Introducing Family Freedom. Our lowest cost will switch our biggest family savings all on America's largest 5G network. Visit your local T Mobile location or learn more@t mobile.com familyfreedom up to $800 per line via virtual prepaid card typically takes 15 days. Free phones via 24 monthly bill credits with finance agreement eg Apple iPhone 16128 gigabyte 82999 eligible trade in eg iPhone 11 Pro for well qualified credits end and balance due if you pay off early or cancel contact T Mobile this.

[Leo Laporte]

Episode brought to you by Red Canary. When cybersecurity threats hit fast, you need an MDR partner that moves faster. Red Canary delivers 24.7expert MDR support, total visibility and actionable insights. Plus it helps you detect four times more threats so you can stay ahead without burning out. Red Canary clears the noise and has your back every hour, every incident. Get the backup you deserve. Visit redcanary.com difference to learn more podcasts.

[Steve Gibson]

You love from people you Trust.

[Leo Laporte]

This is TWiT. This is Security now with Steve Gibson. Episode 1035 recorded Tuesday, July 22, 2025. Cloudflare's 1.1.1.1 outage. It's time for Security now. The show where we cover your security and your privacy Online. We talk about all the exciting stuff happening in the world with this guy right here, Mr. Steve Gibson. A show we wait all Tuesday for. Hello, Mr. Steve Areno.

[Steve Gibson]

All Tuesday.

[Leo Laporte]

All week for Tuesday.

[Steve Gibson]

Oh, okay. There we go.

[Leo Laporte]

That's better.

[Steve Gibson]

From the end of the previous.

[Leo Laporte]

I waited. I wait all day for this show.

[Steve Gibson]

That's right, you're busy all morning before this show.

[Leo Laporte]

No, I wait. I do look forward to it all week because you always find some interesting stuff. In fact, this week I even sent you some things. I don't know if you use them, but I sent you some things to talk about.

[Steve Gibson]

You did? And they're going to need. I'm, I'm, I've been aware of homomorphic encryption for quite a while.

[Leo Laporte]

I'm sure you're the first person ever mentioned it in my presence.

[Steve Gibson]

But mind scrambling technology, the fact that you're able to operate on data without decrypting it, that's crazy.

[Leo Laporte]

And the ultimate in privacy. But we're not there yet because it's computationally a little intensive.

[Steve Gibson]

Does keep its secrets. Yes. Actually, a number of our listeners, like in a flurry, sent me news of today's topic, which is what's really interesting, Leo, is this occurred during the, the hour, the latter part of the hour exactly during last week's podcast recording.

[Leo Laporte]

Oh, interesting.

[Steve Gibson]

Which is that, that due to someone tripping over a cord at Cloudflare headquarters, their famous 1.1.1.1 DNS server service disappeared for an hour globally. So what happened? How does that happen? We're going to have fun talking about that today. Cloudflare's 1.1.1.1 outage for security now number 1035 for this July 22nd. But we got a bunch of other fun stuff to talk about. We've got an active attack successfully bypassing all passkey protections, as in, you know, the FIDO2Pass key ransomware attacks just keep coming. Cloudflare capitulates to the mpa, the Motion Pictures association, and starts blocking, which they've resisted strongly until now. You know, on the whole net neutrality side also, we're going to look at actually, in two different instances, this issue of age verification, which turns out to like be for some reason my main focus. I guess I just see this as like a huge need, an obvious need. And it's like the industry's been caught flat footed because we're, we don't, we have no way to do that. And, and I don't see how we can do it for no cost, which is upsetting because it sort of de. Democratizes the Internet, which every, you know, everyone's been keep, you know, fighting To. To keep open. Anyway, we're going to talk about that a few more times. Microsoft trying to push people from a purchase to a subscription of their enterprise of their exchange servers. Russia further clamping down on their Internet usage and how unfortunately, we begin to see a global trend toward that emerging. China inspecting locked Android phones. So maybe get yourself a burner. Web shells becoming the new buffer overflow. As I said, some. I'm going to sketch out a few different aspects of this whole age verification problem and like play with some protocols that might work. And then we're going to talk about what cloudflare did to create their own complete global massive DNS outage of their flagship server. DNS server for the hour during which we were recording this podcast last week. And of course, as always, we have a great picture of the week. So yeah, I think this one may have been worth waiting for.

[Leo Laporte]

Yay. Absolutely. As always, we'll save homomorphic encryption for another day.

[Steve Gibson]

It's.

[Leo Laporte]

Yeah, it's a hairy topic.

[Steve Gibson]

It's a. Yeah, you lose your hair by the time you've covered that topic.

[Leo Laporte]

It's really funny how much of the time that I'm, you know, I spend every hours, every day reading all the tech news. So I'm preparing for all the shows and how many stories I come across where. Oh, I gotta ask Steve about this. I don't pester you with those. So just to let you know, every once in a while I send Steve one, but I don't.

[Steve Gibson]

What? One escapes the filter. Yes.

[Leo Laporte]

Well, one just came in that I'm a little devastated about. You know, I've talked before about this AI device that I wear, the Bag computer. It's been recording everything that I've done for the last six months and summarizing it in AI and giving me a. They just got sold to Amazon, so I just now deleted all the data. And I'm really hoping the company lives up to its promise to delete all that data.

[Steve Gibson]

So you're wearing it as a lanyard?

[Leo Laporte]

Yeah.

[Steve Gibson]

So it just sort of hangs there?

[Leo Laporte]

Well, I used to wear it on my wrist and then I briefly. I wore it clipped on, but I kept losing it. This is the third one.

[Steve Gibson]

It is audio, so it's not a.

[Leo Laporte]

Camera, it's just audio.

[Steve Gibson]

Okay.

[Leo Laporte]

It records everything. It sends the transcript to AI, which then synopsizes it bullet points. It gives me a kind of a diary, a list of tasks that I. That I think I might want to add to my to do list, which is very, very valuable and generates facts about me over 2,500 facts over the last six months. Things like your wife's name is Lisa, your cat's name is Rosie, you have a Helix mattress, whatever, stuff like that, which is great but I don't want Amazon to have all that stuff.

[Steve Gibson]

So yeah, I'm in the process of setting up a new home and I needed to choose an automation system and I, I, I, I initially thought naively oh well you know, everybody has that A word device, you know, A L E X A. But as I started to play with it I realized I'm part of a big commercial enterprise and I mean they're like upselling me, you know, you know Amazon is, and it's like no, I don't, I'm not using their technology.

[Leo Laporte]

So yeah, I think really we're all just going to end up using Apple's stuff and hope that Apple lives up to its.

[Steve Gibson]

And I settled on, on using HomeKit as the base.

[Leo Laporte]

It's secure, right?

[Steve Gibson]

Yes, we know, we know to the highest levels of the industry that's what Apple has done. I'm a little annoyed with them over their position on age verification because you know, they're so wrapped around that flag that it's like, you know, the industry needs this Apple so but you know, they're doing what they can by, by creating age range brackets and you know, trying to keep it as fuzzy as possible. But anyway, so, so, so you're saying secure? Yes. Private, yes.

[Leo Laporte]

Well we hope so. I mean we're, you somewhat trust Apple but this is so much part of now their marketing that I think they probably will live up to it at least more than Amazon or Google or Microsoft or OpenAI or any of the other possibilities. So it seems like if you're going to do home automation that's the way to go. That's what I'm going to do for sure.

[Steve Gibson]

Did have to choose Google's doorbell because it's the best.

[Leo Laporte]

Well you don't want Ring because Ring just announced that they're going to be sending the information to the cops again.

[Steve Gibson]

So I have a, I have, I have a bridge in order to create that, that, that link.

[Leo Laporte]

So my goal in the long run, and this is a time consuming thing that I'm not going to do anytime soon but is to make it all internal and, and, and you know, use AI internally and all of that stuff. That would be my goal. Same thing with this. I love the idea, the premise of this B computer. I want it to be my own AI, not somebody else.

[Steve Gibson]

Well and, and the problem is the world is switching to a subscription model where it's all services and you. And you know, and so like, you know, you audit your bank account and this is all these little dribbles coming out from all the things that have.

[Leo Laporte]

Happened in the past feels it's almost impossible.

[Steve Gibson]

And you know, as you get older, you really don't want dribbles.

[Leo Laporte]

That's not never good.

[Steve Gibson]

It's not never good.

[Leo Laporte]

Hey, let me talk about another way to secure your enterprise. One of our sponsors for this episode of Security Now, Zscaler. They are the leader in cloud security and they solve a problem kind of in two different directions. You know, AI is, it's a double edged sword, it's a blessing, it's a boon. And it's also perhaps the biggest threat to security we've ever seen. Hackers are using AI now to literally breach your organization. On the other hand, AI powers innovation can drive efficiency. On the other hand, it helps bad actors deliver more relentless and effective attacks. There is a solution. Zscaler Zero Trust Plus AI phishing attacks over encrypted channels last year increased by 34.1%. This year I'm sure it'll even be worse. And that's fueled by the growing use of generative AI tools and phishing as a service kits. We talk about this all the time. And on the other hand, organizations in all industries and small large are using AI internally. They're using it to increase employee productivity. They're using public AIs for engineers with coding assistance. Marketers are using IT for writing. Finance is using AI to create spreadsheet formulas. You ever do a pivot table? Not on my watch. Let the AI do it. You can also automate workflows for operational efficiency across individuals and teams. AI is being embedded into the applications and services that are customer and partner facing. Ultimately, AI lets every company move faster in the market and gain a competitive advantage. But companies, you've really got to think about how you protect your private and public use of AI and at the same time defend against those AI powered attacks. This is what Zscaler does. Jason Kohler, who's the chief Information Security Officer, the CISO at Eaton Corporation, leverages Zscaler to embrace AI innovations and combat AI threats. He says, quote, data loss detection has been very helpful for us. Chat GBT came out, we had no visibility into it. Zscaler was the key solution initially to help us understand who was going to it and what they were uploading. Right. Traditional firewalls, VPNs, public facing IPs expose your attack surface. This old school way of protecting the perimeter is no match to the bad guys in the AI era. It's time for a modern approach with zscalers comprehensive Zero Trust architecture plus AI that ensures safe public AI productivity, protects the integrity of private AI and stops AI powered attacks. It can do all three. Thrive in the AI era with ZSCALER Zero Trust plus AI to stay ahead of the competition and remain resilient even as threats and risks evolve. Learn [e-mail] security that's zscaler.com security security. We thank them so much for their support of security now. All right, Steve. I have not glimpsed, I have not looked. I have not.

[Steve Gibson]

So I gave this, this little cartoon the caption Uncertainty is the nature of the universe.

[Leo Laporte]

Okay. Uncertainty is the nature. I feel like Ed McMahon. Uncertainty is the nature of the universe. I'm going to now scroll up and reveal. That's a cute little cartoon. And you know, it's interesting that the Heisenberg Uncertainty Principle is now so well known that you could actually do a New Yorker cartoon with this and it. Right.

[Steve Gibson]

Yes. It would work and people would get it. Yes. So we have a kind of a professor looking guy staring at the, at, at the map on the wall, trying to figure out, you know, where he, he's trying to go. He is standing. We know this because it says above the map he's, he's in the Heisenberg Department of Physics. And in keeping with the theme, the map has a, the, the legend with an arrow actually pointing to sort of little scatter chart of, of dots. It says you are probably here. So, you know, you can't be certain because you are after all in the Heisenberg Department of Physics. But anyway, it does, it does say.

[Leo Laporte]

Something about how widespread knowledge of quantum mechanics has become, I guess. Yeah, right.

[Steve Gibson]

Yeah. Actually there was a funny. I finished and I'll talk about this a little bit later. I, I finished Project Hail Mary last night and they're reading it.

[Leo Laporte]

We, we should say my reread.

[Steve Gibson]

Yes. And there was one point where, and I don't. I have to be careful not to do any spoilers. Although, you know, the book is. I mean, anyone who's been following the podcast and listening to us and so forth is probably well aware of it. But there was a point where there was a technology exchange and, and, and our, our main character, Dr. Grace, sort of commented that to the person receiving the, the, the human store of knowledge that they're going to be really happy with everything they have received from humanity until they get to the bit about quantum physics because, you know they're not.

[Leo Laporte]

Going to be happy about that.

[Steve Gibson]

That makes nobody happy. It's like, nah, this is just.

[Leo Laporte]

Have we told you about string theory? Oh Lord.

[Steve Gibson]

Yeah. Okay, so the security guys at Expel Security.

[Leo Laporte]

I thought that was a good name. We're gonna expel this.

[Steve Gibson]

Expel Security have uncovered a pass key bypass using, yep, an adversary in the middle attack. Now the vulnerability of pass keys to this attack is actually understood. It's well known. It was a concession that was needed to be made for the sake of cross device login where you're using, you know, a pass key in your phone to log into the. A website on, on a browser somewhere. The Expel security guys just have three bullet points at the top of their blogs TLDR section. They, they said first, bad actors have figured out how to downgrade FIDO key authentication when compromising accounts. Now and we've often talked about downgrade attacks where for example, an early one would have been the client sends to a web client sends to a web server a list of all the protocols it supports. The. But. And normally the web server would deliberately choose the strongest of those offered by the client, which it also supports. Right? So you take like the, you, you, you cross reference the, the, the, the security protocols it has. The security protocols the client has. The server chooses the best, that is the most, the, the, the strongest of those. But if you have an adversary in the middle, the adversary downgrades what the client is sending because of course we haven't established a secure connection at this point. This is the initial client. Hello. So all of the good protocols are stripped out, leading the remote server to believe that the client like supports the paper cup and string protocol, which arguably is not very secure. And so it shrugs its shoulders and establishes a protocol that the man in the middle, the adversary there is able to intercept. So anyway, downgrade protocol. Bad actors have figured out how to downgrade in a similar fashion, FIDO key authentication when compromising accounts. Second bullet point. This technique is being leveraged in phishing attacks, meaning it's happening in the wild that pass keys are being bypassed. Passkey authentication. And finally, the attack involves tricking a user into scanning a QR code with a multi factor authentication authenticator, which includes passkeys. So their blog posting was titled Poison Seed, which is the name they gave this downgrading FIDO key authentications to fetch user accounts. And they explain our soc, which is, you know, abbreviation for Security Operations center has recently spotted a novel attack technique that involves socially engineering a target to get around the security protections provided by Fido passkeys. The attacker does this by taking advantage of cross device sign in features available with Fido passkeys. These features are designed to help users sign into their accounts on systems without a passkey by using an additional registered device like a mobile phone. However, the bad actors in this case are using this feature in adversary in the middle attacks. This is a concerning development given that Fido passkeys are often regarded as one of the pinnacles of of secure multi factor authentication. And while we haven't uncovered a vulnerability in Fido keys, IT and SecOps folks will want to sit up and take notice. This attack demonstrates how a bad actor could run an end route around uninstalled Fido key. We have reason to believe that this attack was carried out by Poison Seed. Oh, that's the name of the group, not the attack. An attack group known for large scale phishing campaigns designed to steal cryptocurrency from their targets wallets. However, the technique described here could easily be leveraged in other attacks. And then they take us through the details of the attack by explaining they said the attack started with a and this is one that actually one of their client accounts was hit by. So they were able to get in there and reverse engineer what happened. They said the attack started with a phishing email sent to several employees at the company. The email lured these users to log into a fake sign in page hosted@okta login hyphen request.com they said this page mimicked the general look and feel of the company's normal authentication process, including an octalog logo and sign in fields for username and password. However, not only is the domain hosting this fake login page suspicious, the domain itself had only been created a week before the attack. Now, I'm going to just pause here to say it's interesting that they provide that bit of detail. We've recently noted that the registration age of any domain a user is visiting or like fetching, you know, for any for whatever reason, should always be raised as a massive red flag. At the very least, any visit to a freshly minted domain ought to be brought to the user's attention. I mean, maybe it's asking too much for many users, but if nothing else, it's an additional signal, right? A a a a sanity check. You know, maybe our our web browsers or an add on or perhaps I.

[Leo Laporte]

Mentioned this before, my next DNS actually blocks it.

[Steve Gibson]

Yes, yeah, yes, Right, right. That if, if it's too new, it's like, no, you need to, you know, prove you, you need to acquire a reputation, but before it'll be allowed. So, as I wrote here, highly security conscious DNS resolvers like NEXT DNS should be checking the age of any domain names being resolved before they're visited. Or perhaps, you know, maybe the page should be displayed. What I was thinking in terms of the browser, the page could be displayed and the user could begin filling out any forms while the reputation is checked in the background. And the form submit function would only be unlocked once a domain reputation, including the domain's age, passed scrutiny. A user could always bypass such a block. But bringing this to their attention and saying, just so you know, you're submitting this to a domain that's only a week old. So does that surprise you? If so, don't proceed. So anyway, I think, you know, somehow, except in your case, Leo, I'm glad that NEXT DNS does this, because this is the kind of thing that responsible DNS providers ought to offer, at least as an option anyway. So they said both this domain and the aws us 3 hyphen manage prod.com domain the user is redirected to if they enter their credentials are hosted by Cloudflare. They said leveraging reputable services like Cloudflare. This is not Cloudflare's fault. Right? They're just the hosting provider can make phishing scams appear more trustworthy, potentially lulling visitors into a false sense of security. They said the targeted user in this case had a Fido key registered to secure their account. Normally, the user would be required to physically interact with the Fido key, touching it, for example, to confirm they're the ones logging in and are on the registered device or using a passkey app. If a user whose account is protected by a Fido key, in this case enters their username and password into the phishing page, their credentials, that username and password will be stolen, just as with any other user, but with a Fido key protecting their account, the attackers are unable to physically interact with the second form of authentication. This is where things took a turn. They wrote from your traditional phishing site. After entering their username and password on the phishing site, the user was presented. The user on the phishing site was presented with a QR code and I have a picture of it in the show notes. It shows iPhone, iPad or Android device. Scan this QR code with the device that has the passkey four and then they blanked it out, but it would be the name of the site. This request comes from the app MSTSC EXE by Microsoft Corporation. And then there's a chunk of the QR code, but they blanked out a bunch so that wouldn't be legitimate. So they wrote what happened behind the scenes is the phishing site automatically sent the stolen username and password to the legitimate login portal of the organization, along with a request to utilize the cross device sign in feature of Fido keys. The login portal then displayed a QR code. Under normal circumstances, when a user wants to sign into their account, they wrote from a different unregistered device. They can still verify their identity if they've enrolled another authentication device. In most cases they this would be an MFA authentication app installed on a mobile device, most of which include a QR code scanner. The login portal displays a QR code after it receives the correct username and password, which the user scans with our MFA authenticator. The login portal and the MFA authenticator communicate to verify the login and the user is granted access. In the case of this attack, the bad actors have entered the correct username and password which they got from the user and entering it into the phishing page, requested cross device sign in the login portal. The legitimate login portal displays a QR code which the phishing site immediately captures and relays back to the user on the fake site. The user scans it with their MFA authenticator. The login portal and the MFA authenticator communicate and the attackers are in. In other words, it is a traditional phishing style attacker in the middle, man in the middle interception, which also includes the optically communicated QR code with the passkey authenticator. The bad guys capture that and they log in as the user, they said. This process, while seemingly complicated, effectively neutralizes any protections that a Fido key grants and gives the attackers access to the compromised user's account, including access to any application sensitive documents and tools such access provides.

[Leo Laporte]

And of course, Squirrel was not vulnerable to this.

[Steve Gibson]

Well, it's an acknowledged weakness of any cross device authentication. Sqrl had this weakness. I went to extreme lengths to eliminate this possibility from any same device authentication. If you used Squirrel's on device app, which was available for all platforms, then any possible man in the middle was excluded because the user's browser connected to the local Squirrel client which performed the authentication. It then received the logged in URL. That is the Squirrel client received the logged in URL which it forwarded to the user's browser. So that cut out Explicitly cut out any man in the middle. They were never able to obtain that logged in URL. But this protection depends upon having a link between the user's browser and the authenticator. And that's not available for typical cross device authentication. The device is able to see the QR code, but there's no way for the device to get a secret back to the user's browser. So it authenticates to the site, which then authenticates the browser session which is logged into the site, which in this case is the phishers, the phishing session, not the phished user session. So I wanted to share this with everyone so that this danger would be very clear. You know, passkeys are a huge step forward and they can prevent many other forms of abuse. If not, maybe it's all other forms, you know, except real wanton negligence. But a determined attacker in the middle that's able to engineer a spoofed phishing attack and convince a user to enter their valid username and password into a site, then intercept and forward their QR code. For a cross device, passkey authentication can still get themselves authenticated even with passkey protected authentication. So you know all the other things that can go wrong with usernames and passwords, you know, passkeys resolve. But it's that cross device authentication which is still the Achilles heel, which is why you really do want to put pass keys into the device where you're doing the authentication. The cross device can be useful for bootstrapping. And as we know, the good news is the FIDO group have now ratified a cross ecosystem. Pass keys, import and export and Apple, they're going to support it and I'm sure everybody else is going to. And we know that the browser add on guys like bit warden and 1Password and so forth, they'll be supporting it too. So it's going to be possible to solve this startup problem that we've had for the first couple of years of passkeys. Cross device authentication cannot be as safe as on device authentication. So that's what you want to use whenever you can.

[Leo Laporte]

Yeah, I'm disappointed.

[Steve Gibson]

Yeah. And boy, Leo, I spent so much time working with the group that I was, that I was interacting with during this that you know, we solved the problem that we solved this completely. If you ran a squirrel client in the device you were authenticating on, but there's, there was just no way to, I mean if you, if you had a, if you had a camera on the machine with the browser, then the URL received the authenticated URL could have been displayed as a QR code on your phone, which you would then show to your browser, to the camera that the browser had and that that could link back. So, I mean, there were some clunky ways to do it. If you had NFC or if you had common WI fi or if you had Bluetooth. All of those things are very messy, you know, they're just not, you know, zero configuration solutions. And what you really want is not to have this raise the bar of complexity to a, to a level where people are like, I don't understand what's happening. I'm just going to use my username because this is too. And it's of course, failure prone too. The more, the more other communication channels you need to have, the, the greater chance for one of them failing. So. Or, you know, getting hacked again. So, yeah, still have a problem. Yeah. For fear of allowing one of the biggest continuing problems the cyber security, the cyber community still faces with his ransomware attacks, I wanted to quickly note a couple of recent biggies. South Korea's largest insurance company, Seoul Guarantee Insurance, got hit by ransomware last Monday. The incident has severely disrupted the company's operations. And the company has been issuing handwritten loan guarantees to customers all week as it works to restore its affected systems. And this is the third major South Korean company to experience severe business disruptions this year due to, due to cyber attacks. The country's largest telecom and its largest online bookstore both suffered similar disruptions. Also, the grocery distributor United Natural Foods has announced that they expect to lose up to $400 million in sales this year following a ransomware attack last month, which took multiple systems offline for days. That downtime affected their ability to fulfill and distribute customer orders. Meanwhile, Australian airline Qantas, you know, the big famous Australian airline, obtained. This is odd. Obtained an injunction to prevent individuals and organizations from using or publishing data stolen from them in a recent ransomware attack. Okay, that's a new one. You know, since when do some foreign bad actors care about an Australian court order? You know, the, the injunction suggests that the company is not willing to pay the ransom and is therefore expecting the hackers to leak the data. Or maybe they just want to protect themselves from that leakage in any event, but it's difficult to see what they expect to gain. Like, okay, we have a, we've got an injunction to prevent any, anyone from using the data that's been stolen from us. Except that criminals stole it from you. And I don't think the criminals care if you have an injunction against them. You Know, in Russia or wherever they may be. So anyway, I just want to just keep everybody aware that like it's business as usual. Unfortunately, in the cyber attack and ransomware world, as we've noted before, there's not even any sign that we are making any progress and improving our effective security. It's just this is now like a steady state, constant background pain that, that companies that are online are suffering when bad guys get in. And you know, as we, we've talked about it before, right. Employees, unfortunately, in large organizations are the weakest link. As far back as that, that Sony Advanced Persistent Threat hack, What was it, 15 years ago or something? I said I sure wouldn't want that responsibility of like, you know, having every employee never make a mistake, never make the mistake of clicking on a link in email. How do you do that? You know, it, the, the problem is it still is a weakness, right? It's an effective problem. Wow. In what's being called a significant turn of events, Cloudflare appears to have changed their long standing policy of enforcing total net neutrality, at least within the uk. The Torrent Freak site, which covers these sorts of net neutrality related events, because that's kind of their focus in general, writes Cloudflare, has become the first Internet intermediary Beyond local residential ISPs to block access to pirate sites. In the UK, users attempting to access certain pirate sites are greeted with cloud flares. Error 451 unavailable for legal reasons. Oh, intercept page.

[Leo Laporte]

Weird.

[Steve Gibson]

Yeah, there actually is. HTTP 451 is. We all know the 404, right? There's a, there's a 451 unavailable for legal reasons is an official, you know, IETF error return. So they wrote. In theory, ISP blocking should prevent UK users from even seeing this notice. But a combination of Cloudflare's blocking mechanism and choices made by some VPN users results in a piracy dead end. Okay, so just to clarify what they mean when they write ISP blocking should prevent UK users from even seeing this notice. What they mean is that since ISPs are the entity that connects UK users to the Internet, ISP blocking should prevent users from ever connecting, from ever even being able to connect to a pirate website. That is, you know, Cloudflare is the host of these pirate sites. And so you shouldn't be. An ISP should prevent their own users. And the isp, of course, is a UK isp. So they're following UK court orders, they're, they're abiding by a block list. And so users shouldn't get to the host and in this case Cloudflare is the host, the article continues. Internet service providers BT, Virgin Media, Sky Talk, Talk, EE and Plusnet account for the majority of the UK's residential Internet market. And as a result blocking injunctions previously obtained at the High Court often list these companies as respondents. These so called no fault injunctions stopped being adversarial a long time ago. Right, where ISPs would like say no, we're not going to do that. Torrent fried wrote that ISPs indicate in advance they will not contest a blocking order against various pirate sites and typically that's good enough for the court to then issue an order with which they subsequently comply. So everybody's just getting along with this now, they wrote. For more than 15 years this has led to blocking being carried out as close to users, meaning their ISP the so called last mile as possible. With ISPs individual blocking measures doing the heavy lifting, a new wave of blocking targeting around 200 pirate site domains came into force last Monday the 14th. But with the unexpected involvement, they wrote, of a significant new player. In the latest wave of blocking that came into force, close to 200 pirate domains requested by the Motion Picture association were added to what was already one of the longest pirate site blocking lists in the world. The big change is the unexpected involvement of Cloudflare, which for some users attempting to access the domains added yesterday displays the following notice and I've got it here in the show notes. It's A big error HTTP 451 with a a a time code date.

[Leo Laporte]

Now I know what that is.

[Steve Gibson]

Yep, yeah. Available for legal reasons and then under, and then as an explanation under what happened, they wrote in response to a legal order, Cloudflare has taken steps to limit access to this website through Cloudflare's pass through security and CDN services within the United Kingdom. Oh and, and I'm sorry and CDN services within the United Kingdom. Find more information about the order, the party that requested it and the authority that issued it here. And the here was lit up in blue as a link that that users who received this intercept page could click on and they said learn more about Cloudflare's approach to blocking orders in our transparency report on abuse processes and then another link. So when we've previously covered this issue and we've applauded Cloudflare's adamant pro net neutrality stance, we've cited Cloudflare's formal policy statement about this and this may sound familiar to our listeners because it reads because this is Cloudflare speaking, because Cloudflare cannot remove content, or because Cloudflare cannot remove content it does not host, other service providers are better positioned to address these issues. Among other things, any blocking by Cloudflare is of limited effectiveness as a website will be accept accessible if it stops using Cloudflare's network. Cloudflare therefore regularly pushes back against attempts to seek blocking orders. So, so the point they're making here is that they're sort of deflecting. They're saying if an ISP, they're, they're saying users, local ISPs are the correct and better place for blocking enforcement. Because if, if somebody, if a pirate relocated from Cloudflare to anywhere else, well, the ISP would still block their domain. It doesn't matter who's hosting their domain, the domain is blocked. So Cloudflare is saying, whereas if you tell Cloudflare to block a domain, well, they can move to a different host where the domain would not be blocked. So this has always been Cloudflare's formal position. You know, they're, they're saying, you know, don't ask us to take responsibility for the content we're hosting because there are many other hosting providers. So Torrent Freak explains. Cloudflare notes that it may take steps to comply with valid orders if, among other things, that is in this new HTTP 451 intercept page, if, among other things, quote principles relating to proportionality, due process and transparency are upheld, they wrote. Whether Cloudflare pushed back here isn't clear, but the information made available turns out falls well short of that promised in the Error 451 notice, that is. Torrent Freak followed those links and felt very unsatisfied by the disclosure that Cloudflare was offering by those links, they said. With no central repository for blocking orders and no legal requirement to share details of injunctions with the public, transparency in the UK is mostly left to chance. Some orders make their way online, but there's no guarantee. For those interested in finding out more about the order affecting Cloudflare, the company provides a link which promises to reveal, quote, the party that requested it and the authority that issued it. The link directs to the Lumen database, which publishes information effectively donated by companies such as Google and Cloudflare for the purpose of improving transparency. But in this case, there's no indication of who requested the blocking order or the authority that issued it. However, from experience, we know that the request was made by the studios of the Motion Picture association and for the same reason the high court in London was the issuing authority. To the general public, the information is just a short list of domains. If it wasn't for the efforts of Lumen, Google and Cloudflare volunteering, the situation would be significantly less clear than that. So Torrent Freak here is noting and complaining that there's a real problem with a lack of transparency and accountability with the way the system is working now. They add the issue lies with dynamic injunctions. While a list of domains will appear in the original order, which may or may not ever be made available, when the MPA concludes that other domains that appear subsequently are linked to the same order, those can be blocked too, but the details are only rarely made public from information obtained independently. One candidate is an original order obtained in December of 2022 which requested blocking of domains with well known private pirate brands, including 1, 2, 3 movies, F movies, Soap Today, Hura watch and S Flicks and Onion Play. And they finished. What's odd is that the notice linked from Cloudflare doesn't directly concern Cloudflare. The studio sent the notice to Google after Google agreed to voluntarily remove those domains from its search indexes if it was provided with a copy of relevant court orders. Notices like these were supplied and the domains were de indexed and the practice has continued ever since. That raises questions about the nature of Cloudflare's involvement here, and why it links to the order sent to Google. Notices sent to Cloudflare are usually submitted to Lumen by Cloudflare itself. That doesn't appear to be the case here. So they're just sort of generically unsatisfied with with Cloudflare's lack of transparency. Maybe just that this all happened very quickly. You know, again, we don't know what's going on, but what's interesting is that Cloudflare is Now putting up HTTP, you know, 4, 51 responses saying and blocking their own clients access to their own clients. Oh, and as for VPN circumvention Tort Torrent Freak wrote, when blocking measures are required, Cloudflare digs in when requests concern its public DNS resolver, which we will be spending more time talking about at the end of the podcast, you know, 1.1.1.1 they wrote. To achieve a similar effect, Cloudflare uses another technique instead. Okay, so I assume they mean that Cloudflare still adamantly refuses to muck up their master public DNS resolvers with filters. And thank God for that.

[Leo Laporte]

Yeah, rightly so. Yes, we've seen horrible things happen in the past when the the international Soccer broadcasters blocked.

[Steve Gibson]

Right.

[Leo Laporte]

The DNS. It screwed up everything.

[Steve Gibson]

Yes, we know. We, we need a a strong universal DNS resolver that is not subject to the whims and needs of any particular industry or government. And by the way, Cloudflare does offer 1.1.1.2 and 1.1.1.3, which are two is family friendly filtering and or no, okay, one is malware only one is malware and family friendly or one is family friendly and one is malware. Anyway, I know what they are and they're now part of the DNS benchmark, but I don't have it off the top of my head. So Cloudflare is offering some filtering services, but if you want the, you know, an industry standard, absolutely clean DNS, I'm really glad to hear that they're saying no, we are. We are not going to start screwing around with with our DNS. The torrent Freak quoted Cloudflare saying in countries with laws that provide for blocking access to online content Cloudflare mage this is a Cloudflare speaking and which Torrent Freak quoted Cloudflare may geo block websites to limit access in the relevant jurisdiction to those websites through Cloudflare's pass through security and CDN services. Okay, so in other words, Cloudflare is not going to filter DNS, but they will, when required by law. Tipping the typically taking the form of court orders, filter access to some list of sites they host based upon the location of the client making the connection to Cloudflare's network. So they're able to do, as they said, geo blocking, Torrent Freak wrote. Cloudflare appears to be using geo blocking in the uk, as some VPN users will soon find out. In normal circumstances, they said, a VPN using a server in the UK will bypass ISP blocking no differently than a server located anywhere else in the world. Users attempting to gain access to domains currently blocked by Cloudflare using a VPN server in the UK will be greeted by Cloudflare's Error 451 blocking notice instead. So what they're saying here is that whereas in the past basically any VPN would have been useful in jumping past that local a user's local ISPs block and it didn't matter where the VPN was terminated by its server. Now that Cloud Cloudflare is implementing UK wide geoblocking of content above and beyond what ISPs may also be doing, any UK based VPN users will need to be sure that they're terminating their VPNs at servers outside the UK, which would not then be blocked by Cloudflare, and that would have not been previously necessary when only their ISP was doing the blocking. And they concluded by noting that the scale of this blocking appears to be large. Writing checking through the new domains blocked on the 14th, something else becomes apparent. They appear in multiple blocking orders, not just the ones highlighted in their article. They said, we're un, we're unable to check all 200 domains, but at least potentially hundreds or even thousands of domains could be involved. And that may actually be a very good thing. I thought. What, okay. They said domains blocked by Sky, BPI and others don't appear to be affected, at least as far as we can determine. All relate to sites targeted by the mpa. And the majority, if not all, trigger malware warnings of a very serious kind, either immediately upon visiting the sites or, or shortly after, at least in the short term. If Cloudflare is blocking a domain in the uk, moving on is strongly advised. So I believe they're saying that the blocking Cloudflare has begun doing appears to relate to domains hosting malware, perhaps more than just those that the MPA may be grumbling about. You know, so I could then see where Cloudflare is like 100 behind, you know, blocking of malicious websites or access to, to them. That, that seems like a lower bar than, than getting into a, an argument over copyrights. So whatever the case, it appears that Cloudflare is simply doing, you know, they're abiding by the low, by, by the law, you know, as they're required to if they're going to operate in, in the uk. Although it's a little sad or disappointing because I'm such a Cloudflare fanboy. Overall, the evidence is that, you know, Cloudflare doesn't seem to be explaining very clearly exactly what they're doing and why, at least not in the notice that that torrent freak pulled up and, and looked at that they received last week. Maybe it's just, it just happened.

[Leo Laporte]

They're in a tough business. I mean, they're really in a tough situation.

[Steve Gibson]

Yes, and, and you know, we, I'll be talking a little bit more about this later, but we're seeing more and more of this where governments are increasingly mucking around in what used to be just a hands off, you know, fully democratized Internet.

[Leo Laporte]

That's only because it was under the radar. I mean, as soon as it became mainstream, they said we gotta control this.

[Steve Gibson]

Yep, that's exactly what I conclude, Leo, is that, you know, we were all having fun before it mattered.

[Leo Laporte]

Exactly this portion of security now brought to you by one Password. Love these guys. You know, this was kind of a stunning stat. Over half of IT professionals, you know, these guys who secure your business say that securing SaaS apps is their biggest challenge. With the growing problem of SaaS sprawl and shadow it, it's not hard to see why. Thankfully, there's a solution. Trelica by 1Password Trelica T R E L I C A can discover and secure access to all your apps, whether they're managed or not. Trellica by 1Password inventories every app in use at your company. Then pre populated app Profiles assess the SaaS risks letting you manage access, optimize, spend and enforce security best practices across every app your employees use, whether you know about it or not. But now you will know about it, right? You'll be able to manage shadow it. Oh, it also lets you securely onboard and offboard employees and meet compliance goals. So it it really is a great tool. Trelica by1Password provides a complete solution for SaaS access and governance. And it's just one of the ways that extended Access management for 1Password helps teams strengthen compliance and security. 1Password's award winning password manager is trusted by millions of users and over 150,000 businesses from IBM to Slack. And now they're securing more than just passwords with 1Password Extended Access Management. 1Password is ISO 27001 certified. With regular third party audits and the industry's largest bug bounty, 1Password exceeds the standard set by various authorities and is a leader in security. So take the first step to better security for your team by securing credentials and protecting every application, even unmanaged. Shadow it. Learn [e-mail] SecurityNow that's 1Password.com Security now, all lowercase. Thank you 1Password for supporting security now and the good work Steve Gibson's doing here. Kind of similar to what you're doing with 1Password Extended Access. Now back to Mr. Gibson.

[Steve Gibson]

So the heat surrounding Internet user age verification continues to increase. I'm encountering an increasing level of pressure in like just more and more in the various news items that I survey. For example, last Thursday Roblox posted an update which included this under the headline Age Estimation. They said Roblox is investing in is investing an age and I think they meant investigating or investing in not clear an age estimation technology to help deliver tailored and developmentally appropriate experiences while aiming to protect its community from those who might seek to do harm. To add contacts as a Trusted connection users must be 13 and over and confirm their age using a video selfie, which is analyzed against a large, diverse data set to estimate their age. Matt Kaufman, Roblox chief safety Officer, said, quote, we know teens want more freedom to chat more freely with their friends. We believe that unfiltered chat should only be made available to users who have been age checked, which is why we're using new age estimation tools to unlock access to trusted connections for those 13 and over. We believe this additional freedom to chat more openly will reduce the incentive for teens to move interactions off platform where they may be exposed to greater risk, unquote. So, okay, you know, how, I don't know, subject to maybe tampering or spoofing or whatever, facial recognition could be trying to guess someone's age. I mean, I'm not a teenager, no one's going to confuse me for that. But still, you know, down in the, you know, 13 or so pretty young.

[Leo Laporte]

You could pass for. You could imagine 65 at least. You know, give me a wig maybe.

[Steve Gibson]

Yeah. So, okay, there's that. Steam reports that they're being pressured, you'll pardon the pun, over some of their content by the payment processors, believe it or not, that they use in response, rather than risk losing their payment flows. Steam has reportedly removed thousands of games containing adult content, though what that is remains unclear. Like, that's, like what exactly the content should be. Last Friday, Eurogamer asked Valve for some clarification and then wrote this of their response. They said in response to questions from Eurogamer regarding Steam's new guidelines preventing, quote, certain types of adult content, unquote, from being distributed on the platform. Valve has provided some general background on the events leading to the decision. A Valve spokesperson told Eurogamer, quote, we were recently notified that certain games on Steam may violate the rules and standards set forth by our payment processors and their related card networks and banks. As a result, we are retiring those games from being sold on the Steam store because loss of payment methods would prevent customers from being able to purchase other titles and game content on Steam. So, okay, in this case, thousands of titles are being removed without regard for the age of the user in what appears to be a case of I'm looks like blackmail, censorship by Valve's payment providers. So I'm sure it must be clear to everyone by now that the need to verify the age of Internet users is not off. Someday in the future, you know, we need the W3C or the IETF or perhaps the FIDO Alliance. You know, if any of Them could move at anything other than glacial speed to get busy, go and whip up some standards because we need some technology here, you know, then we need Google and, and Apple to implement them in their biometrically equipped devices. And, and, and my concern is that these things are so expensive, you know, these high end smartphones that there would be a place for someone like a next generation yo you know, to create cute, inexpensive little spoof resistant thumbprint authenticators that would follow the same specification, which unfortunately doesn't yet exist. You know, and we need all of that yesterday because the need for age verification is today. So you know, imagine that a, a Yubico type thumbprint sensor age verification verifier existed. If you, if you have a biometrically lockable smartphone, then you wouldn't need an extra gadget because the phone you've got would be able to do that. But as I said, my concern is that such smartphones are very expensive. So we need a, you know, a 20, 30, $40 alternative. If you don't have some suitably equipped smartphone, you buy an inexpensive gadget from a local retailer, a neighborhood electronics, you know, store outlet, whatever. So in, in my little thought experiment here, how do we, how do we arrange to create the binding between the user's biometric and an assertion of their age? And how do we do it at scale? Someone who wishes to enroll their iPhone, their biometric Android device or some inexpensive theoretical thumbprint verifier takes their chosen device to any US post office. They, in the US the DMV maybe AAA if you have a membership or any notary like is available at any UPS store in, in, in the United States. You show them your government issued ID proving your age, you know, they check it carefully for forgery, you know, look at, you look at your ID and then have the user in front of them authenticate with their chosen biometric, you know, their face or their thumbprint depending upon their device, after which the agent uses their own device, any NFC equipped phone or terminal or, or Bluetooth or whatever to essentially bless and activate and lock that biometric to age binding. Now this individual is in the possession of a biometrically locked age assertion which they can use on demand anywhere in cyberspace that it's needed. Again, we don't have the protocols, we don't have as far as I know, any little, well there is no protocol, so there's nothing for anybody to implement, you know, on any platform at this point. But you know, there's a, you know, a little bit of brainstorming about how we might begin to solve this problem. And this is no, it's a good thing, Leo, that I'm very committed right now to the projects that I have in front of me because you know, this is pulling me in the same way that squirrel pulled me 10 years ago. And we know how that went and went seven years of my life. But anyway, it seems to me this is like so necessary. A bit later in today's podcast, in answer to one of our listeners questions, I'm going to sketch out an example of a cryptographic protocol to provide again just a rough sense of for some more of the details of this. But my overall point is that this, the problem is not intractable, but it's not easy either and people need to get moving on this and I don't see any sign of this happening. You know, even though Yubico's founder Stina Arensford has moved on to other passions. I dropped her a note yesterday as I was writing this. You know, she's the perfect kind of person to, to shake things up and get the industry's attention and get this moving. I did get email back from her. I, I found it waiting for me this morning saying that she does have a not she's established a non profit which is, it doesn't seem focused on age but she's still on the identity crusade. She did you know I did tell her about my, my, my concern over the need for some sort of workable privacy respecting age verification and she said that that's what she was doing and wanted to set up a conference and see how we could collaborate. And I again I don't want to get too sucked into something because I've got work to do. But this just to me this seems like like one of the biggest needs we have because the world is starting to wake up to the Internet it seems and the age of the people using it is suddenly a big deal. So we need protocols. I hope somewhere that's beginning to happen. In other news, it appears that Microsoft remains unsure what to do about the fact that that no one appears to actually want their new crap. Especially in light of the fact that Exchange Server in this case is switching to a subscription. What a surprise. You know, I guess you no one should be surprised that no one is in a big hurry to switch to subscription mode. Everyone wants to just keep using the stuff they already have that's working just as well as any of the new stuff probably would, especially when they already paid for the stuff that they have that's all installed and running and configured and working just fine. So in this case, we're talking about Exchange 2016 and 2019 server, whose end of life is scheduled for that same fateful day approaching us on October 14, when Windows 10 and some other Microsoft products that no one wants to be forced to stop using were originally scheduled to stop receiving their security updates. But because users of Exchange Server are not just some rando consumers, you know, anyone who has so far refused to jump at the opportunity to switch to their marvelous new pay as you go subscription plan for Exchange Server is going to need to pay up. And Microsoft says that's it. We're serious this time. No, really, no kidding. This is it. You're actually going to have to, you know, do this. They actually, they actually wrote don't even bother asking for more. So last Tuesday's Exchange Team blog posting under the headline announcing Exchange 20162019 Extended Security Update program, they wrote, with both Exchange 2016 and 2019 going out of support in October 2025. We've heard, I bet they have. We've heard from some of our customers that they've started their migrations to Exchange Subscription Edition. Literally, it's SE for Exchange Subscription Edition, but might need a few extra months of security updates for their Exchange 20162019 servers while they're finalizing their migrations. We are announcing that we now have a solution for such customers starting on August 1, 2025. So the end of this month, August 1 customers can contact their Microsoft account team to get information about and purchase an additional 6 month extended security update ESU for their Exchange 20162019 servers. Your account teams will have information related to per server cost and additional details on how to purchase and receive ESUs starting August 1, 2025. Now logic would suggest, you know, that the, the, the stay right where I am for the next six months plan will cost more than the, you know, that subscription sounds great, sign me up plan. And you know, no one ever accused Microsoft of leaving any money on the table, so it will almost certainly cost those foot draggers more than getting on with the new plan. Microsoft continued writing. So what does this mean? They said first this ESU is not an extension of the support life cycle. And they said Microsoft Lifecycle Policy Microsoft Learn for Exchange 2016 2019. Those servers still go out of support on October 14, 2025 and you will not be able to open support cases for them unless directly related to an issue with a SU released to esu. That is a a service update released to ESU customers during the ESU period. So they said the ESU is not an extension of the support life cycle. Okay, I don't understand why because that's what they're selling you. They said this ESU is a way for customers who might not be able to to finalize their migrations to Exchange SE the subscription edition before October 14th to receive critical and important updates as currently defined by Microsoft Security Resource center scoring as SUS security updates that we might release after October 2025. Okay, so I guess what they're saying is you have to have signed up for the subscription, but you may we understand you may not have yet finished migration to the subscription servers or the subscription from your non subscribed Servers Exchange Server 2016 and 2019 so you can buy additional support for them in order to bridge they said Exchange 20162019 sus these service updates will not be released on Public Download center or Windows update after October 2025. So they're still trying to be as as strict here as they can. They also said we are not committing to actually releasing any service updates during the ESU period, meaning you pay for it and you may not get anything. They said Exchange Server does not necessarily receive security updates every month on Patch Tuesday as security updates are released only if there are critical or important security product changes. Therefore, if there are no issues that we need to release during the time of esu, there will be no SO update, no such updates provided. We will however confirm with ESU participants each Patch Tuesday whether an SU was provided or not. This ESU will be valid they said for six months only through April 14, 2026. And they wrote this period will not be extended past April 2026. You do not need to ask. So anyway, that's the story. If you are an enterprise, you're not going to be ready by October 14th to stop receiving any security updates for your existing Exchange 20162019 servers. Then you can buy any that may occur. I wonder if you could wait to see if any occur and then buy them then. I I don't know. Anyway, they they they finished explaining. They said customers using Exchange 2019 should in place upgrade to Exchange SE quickly and switch to the Exchange SE Modern Life Cycle policy. Meaning yes, the Modern Life Cycle Policy, also known as the will no longer allow you to purchase it in these modern times. You now keep paying for it forever. So anyway, for what it's worth, the the wonder and clever folks over at 0patch. You know it's numeric 0p a t c h.com the 0patch guys do provide patches for Exchange Server and they do so on very reasonable terms. So it might be more cost effective to consider remaining with the already paid for in full Exchange Server you already own and then having the zero patch folks keep it up to date for you. You know, basically they recreate Microsoft's patches, they reverse engineer them and then offer them like they don't even have to reboot Exchange Server. Right? I mean it's like way better than Microsoft until April 14th when that when those older servers will no longer be receiving security updates for the micro patch guys to reverse engineer. And I don't know whether they can look at the security updates for the next generation of Exchange servers and backport them to the earlier editions of Exchange Server. We'll have to see at that point. But don't forget those zero patch guys. They're going to be friends of Windows 10 users also starting October 14th as we talked about before. Okay so wow. A new Russian law has get this LEO criminalized online searches for controversial content. Russia previously criminalized the sharing of such content or obtaining it, but with officials saying that censorship during wartime is justified, that is they're they're using their war with Ukraine as the context here. They're saying restrictive digital laws are justified and being tightened. The Washington Post reported this on last on this last Thursday, writing Russian lawmakers passed controversial legislation Thursday, meaning last Thursday that would dramatically expand the government's ability to punish Internet users not for sharing forbidden content, but for simply looking it up, like putting the search term in. The new measures, which sailed through the Russian Parliament and will take effect in September, envision fining people who, quote, deliberately searched for knowingly extremist materials, unquote, and gained access to them through means such as virtual private networks or VPNs, which lets users bypass government blocks. VPNs are already widely used in Russia to circumvent the many blocks on websites, the Washington Post wrote. Russia defines extremist materials rather broadly as content officially added by a court to a government maintained registry, a running list of about 5,500 entries at the moment, or content produced by extremist organizations ranging from LGBT movement to Al Qaeda. The new law also covers materials that promote alleged Nazi ideology or incite extremist actions. Until now, Russian law stopped short of punishing individuals who for seeking information online, only creating or sharing such content was prohibited. The new amendments follow remarks by high ranking officials that censorship is justified in wartime. Adoption of the measures would mark a significant tightening of Russia's already restrictive digital laws. Similar legislation they wrote, passed recently in neighboring Belarus, Russia's close ally, ruled by authority authoritarian leader Alexander Lukashenko and has been used to justify prosecution of government critics. The fine for searching for banned content in Russia would be about $65, while the penalty for advertising circumvention tools such as VPN services would be steeper $2,500 for individuals and up to $12,800 for companies. Sarkis Darbinian, an Internet freedom activist whom the Russian authorities have labeled a foreign agent, said the fines imposed for searching for extremist materials in this iteration may be minor, but this can be grounds for detention, pressure or a pretext to be escorted to the police station. I am most afraid that in the next iteration, administrative fines will turn into criminal cases. Previously, the most significant expansion of Russia's restrictions on Internet use and freedom of speech occurred shortly after the February 2022 full scale invasion of Ukraine, when sweeping laws criminalized the spread of so called fake news and discrediting the Russian military. The new amendment was introduced Tuesday, attached to a mundane bill on regulating freight companies, according to documents published by Russia's lower house of parliament, the State Duma. We talked about before Net Freedoms, an advocacy group said in a statement. Lawmakers have repeatedly used this cunning tactic of quietly inserting repressive measures into dormant, previously introduced bills. It allows them to accelerate the legislative process, moving through the second and third readings in a single day and to avoid public scrutiny. On Wednesday, as news of the censorship amendment sparked widespread concern in Russian media, lawmakers pushed the bill sought to down pushing. The bill sought to downplay fears that citizens would be penalized for browsing the web. Senator Artem Sheikin, one of the bill's authors, told state controlled news agencies that the new measures are not intended to punish individuals for accessing prohibited websites, using VPNs, reading Facebook or scrolling through Instagram, Sakin said, does not constitute an administrative offense. The main focus is on regulating providers, he said. There's no plan for mass punishment of users. He claimed that liability would only attach in cases of knowingly searching for and accessing content officially designated as extremists by a court and added to a Ministry of justice blacklist. However, he did not explain how authorities would determine whether an individual knew the access content was deemed extremist. Anyway, things are tightening up in Russia and they they they used the term throttling, talking about how Russia is also has also expanded its use of deep packet inspection technologies, enabling more precise blocking of traffic and committed millions of dollars to fortify what we Know, as, you know, Russia Net or RU Net, it's creating this sovereign Internet infrastructure that allows them to pull the switch and disconnect Russia from the rest of the global Internet. They also said that telecom providers have been ordered as, and we talked about this before, to provide detailed user data while citizens are being pressured to use domestic platforms instead of the foreign ones by throttling or restricting platforms such as YouTube X and Instagram as the Russian government seeks to limit access. And you know, we talked about the use of the term throttling because Cloudflare sites were recently added to this throttle technology where a page was limited to 16k bytes if it came from Cloudflare, which as I observed was really not enough to run any, like even begin to get a modern web page off the ground. Maybe you could do a 301 redirect. Well, you could do that in 16K. And that was the only explanation that I could come up with. But as we've said, any site that was, that had content that want from that was of interest to Russians could just move to a Russian hosting provider in order to get around that block, which is probably the whole goal here. So for me, this news is disturbing. I'm not in Russia, but Russia is an extreme example of what we're seeing everywhere. This general tendency globally from the world's governments, the UK and the EU are chafing over encryption and arguing against fundamental privacy rights. Here in the US we've seen the Supreme Court just approve the means by which various extreme special interest groups will be able to, to criminalize, essentially enter any Internet speech that they dislike or deem to be unwholesome. The, the definition of in. In the legislation that the, the U.S. supreme Court just approved is very worrisomely broad. And as I was saying, Leo before, it feels as though for the first 50 years of the Internet, you know, it was not well understood and sort of remained out of bounds for the world's governments and politicians. Or as we noted, perhaps it just didn't matter all that much until just the past decade or so. You know, we enthusiasts were all having a great time playing in our sandboxes with our technologies, but now the political adults have returned and they're scowling at the things that we've been up to. Yeah, I don't know.

[Leo Laporte]

I mean, I don't know what the answer is.

[Steve Gibson]

I mean, it does, it does feel like it's all changing.

[Leo Laporte]

Rapidly.

[Steve Gibson]

Yeah. Let's take another break and then we're going to talk about a bunch of more stuff and good stuff, important, some.

[Leo Laporte]

Listener feedback and of course, 1.1.1.1.

[Steve Gibson]

We're going to get there too.

[Leo Laporte]

More Cloudflare news coming up. Yep, this episode of Security now is brought to you by Acronis. You know that name? We talk about them all the time, especially the Acronis Threat Research Unit. How many times have we quoted them in the security bulletins and information? Well, they can work for you too. You, Dear IT professional, deserve fewer headaches in your life. Even something as simple as watching TV these days can become a headache when your favorite shows are scattered across different streaming services, it's nearly impossible to find one place that has everything you need. Now bear with me. I'm not talking about streaming TV so much as taking the headache out of cybersecurity with Acronis, a natively integrated platform that gives you comprehensive cyber protection in a single console so you don't have to go searching around for the information you need. If you want to know what's happening in cybersecurity, the Acronis Threat Research Unit, the tru, is the place to go. It's your one stop source for cybersecurity research. TRU also helps MSP stop threats before they could damage you or your client's organization. Acronis Threat Research Unit is a dedicated unit composed of experienced cybersecurity experts. Their team includes cross functional experts in cybersecurity, AI threat intelligence. TRU conducts deep, intelligent driven research into emerging cyber threats, proactively manages cyber risks and responds to incidents, plus provides security best practices to help you and your IT team in building robust security frameworks. They also offer threat intelligence reports, custom security recommendations, and educational workshops. If you're listening to Security Now, I know it means you need this kind of information. Well, whether you're an MSP looking to protect clients or you need to safeguard data in your own organization, Acronis has what you need. It's all there in Acronis Cyber Protect Cloud, edr, xdr, Remote monitoring and management, Managed detection and Response, email security, Microsoft 365 security, even security awareness training. And it's all available in a single platform with a single point of control for everything, so it's easy to deploy and manage. If managing cybersecurity gives you a headache, it's time to check out Acronis. Know what's going on in the cybersecurity world by visiting go.acronis.com twit and take the headache out of cybersecurity. That's go.acronis.com TWIT A C R O N I s. You know the name, you know the tru, maybe you got to get them working on your behalf. Go.acronis.com TWIT we thank them so much for their support of security. Now very often we've quoted research from the tru. They're good people. All right, Steve, on we go.

[Steve Gibson]

People may want to pick up a temporary burner Android phone when traveling to China if they are an Android phone user. Turns out that Chinese authorities are using a new forensic toolkit to extract data from Android phones. Yeah, the new tool named Mesistant is being used at border checkpoints and by local police forces. It's able to extract geolocation data, images, SMS messages, contacts and other data from third party messaging apps. According to the mobile security firm Lockout, Massistant appears to be the successor of a previous tool used by authorities named MF Socket. And just as another note, anyone switching to the use of a burner phone should probably begin using it sometime before the trip so that it can accrue some believable history. There have been instances of people, you know, being further harassed when their use of a burner was made obvious by its lack of any extractable historic data. You know, you give the authorities an empty phone and they stare at you and goes, okay, where's the phone you actually use?

[Leo Laporte]

So there's a certain irony in this because same thing happens when you enter the United States if you're a foreign national.

[Steve Gibson]

And actually it's good. It's funny you mentioned that this mailing and the show notes went out yesterday afternoon and I got a note from a listener saying, for what it's worth, the USA is just as bad.

[Leo Laporte]

Right?

[Steve Gibson]

And I actually, I included that in next week's show because I wanted the reality check that. Yes, it's not like our hands are, are completely clean in this either.

[Leo Laporte]

You know, in general, I think if you're going to travel internationally, you need some sort of plausible deniability. Maybe, maybe get a Chromebook and I don't know, and a burner phone and. Wow, it's sad that we have to do that. You know, honestly, I have no plans to travel outside the US for the foreseeable future for that reason.

[Steve Gibson]

Yeah, it's changed.

[Leo Laporte]

It's changed. I love China, by the way. I love visiting China. It's an amazing country.

[Steve Gibson]

Yeah. Yeah. After encountering the following bit of news, it occurred to me that perhaps remote web management access of any kind, regardless of how well authenticated its Designers and deployers certainly believe it to be that it's really risen to the status of the much heralded buffer overflow or overrun. You know, it's just. It tops the list of recurring, ubiquitous, really dumb things to do. The news is that once again security researchers, this time with the Shadow Server foundation have found web shells on that is installed by militia maliciously installed web shells on almost 80 Fortinet forta web firewalls. The Shadow Server foundation believes the web shells were installed after hackers exploited a Recently patched vulnerability CVE2025 2525 7. The bug here it comes again is a pre auth SQL injection in the firewalls web panel. Fortinet has not yet confirmed in the wild exploitation, which I thought was humorous. Apparently they're the last to know since 80 individual instances of a fortinet for the web firewall compromise ought to be pretty easy to confirm. You know, sounds like they just may not want to pull their head out of the sand and be in any big hurry to confirm it officially. But anyway, again, you know, historically it's been buffer overruns. That's been the mistake everybody keeps making. Well, it now looks like that's been supplanted by web portal compromises. We seem unable to put up a web portal whose authentication cannot be bypassed. So of course my conclusion is so don't put them up. Just restrict them in a way that is actually strong and useful rather than relying on a username and password. That's just no bad idea. I. I wanted to mention before we talk about listener feedback, that last night I finished my very pleasurable reread of Andy Weir's project. Harry May, Hail Mary. Good. Yeah. And Leo, I have absolutely no idea how anyone could possibly turn this into a hyper condensed two hour enjoyable movie which is in any sense faithful to the book. I. I would not want to be the screenwriter or the director, you know, we're going to find out next, next March 20, which is its release date. I don't doubt that people who have never read the book, you know, will still love the movie. I think it looks like it's going to be a really fun movie, but the book was really terrific and whatever the movie will be, I can't see how it could possibly be anything but a rough, you know, the roughest of outlines of the events in the book. And I was surprised Lori said that she felt some of the physics was kind of beyond her. She didn't, you know, track it all. I'm okay. I mean, it's all. There's a lot of science in there. But. And I think she actually was being modest. I think she understood most of what was going on. But you know, certainly that a lot of that won't make it into the movie because that would be way past your, your typical audience and probably isn't necessary. I think that's what a, you know, an enthusiast who reads the, who reads the novel wants. But anyway, I did immediately purchase Andy's second novel Artemis and it's now loaded into the five Kindles which I use.

[Leo Laporte]

Why five Kindles?

[Steve Gibson]

Well, I have one Kindle device. I've got it one iPhone and three iPads and I move, I move among them.

[Leo Laporte]

Sure.

[Steve Gibson]

From, from day to, you know, from like does. Yeah, yeah, yeah. So I have one. My iPhone in my pocket. I've got a, a Kindle mini buy at at on the bed, on my, my nightstand next to the bed. I've got one on a big pad downstairs and then one on my Kindle device, the Oasis that I love and I'll take that with me like if I'm gonna be off site somewhere, you know, like of I'm. I'm doing transport for a friend or, or something and I have some, I'll have some time to kill where I can't really do anything else. So anyway I'm gonna read Artemis and I'll let people know what I think.

[Leo Laporte]

Good.

[Steve Gibson]

Okay. Bob Van Metteren said hi Steve, just wanted to write. Oh, that a guy. I love this. A spin right? Level 3 refresh of my 2017 Kindle fixed my issue and he wrote as if I already knew that he had an issue. I couldn't find any reference to any previous feedback from him or writing to support or anything. And he said, he said thank you for this amazing product. He said I'm also a loyal security now listener since 2019 and grew up with a speak and spell. Oh yeah, that's, that's the speaking spell right there behind me. That orange thing. Yeah. He said so thanks for that too because I was involved in its development and he said we can infer anyway so thank you Bob. We can infer from his note that he has an 8 year old Amazon Kindle that developed some sort of problem. Of course during the 3 and a half year Spinrite 6.1 development we learned much more than we knew there was to learn about the surprising age related decline in the performance and reliability which are closely related of solid state storage. We also learned that Spinrite's ability to recover data that's become marginal, coupled with its rewriting of solid state data more often than not completely reverses this decline and rejuvenates storage. As an avid Kindle owner myself who often exports books from the device for archiving, I am well aware that connecting a Kindle to a PC allows the PC to view the Kindle's storage as a solid state drive. And that's all spinrite needs to be able to work its magic on any device such as a Kindle. You know, we sometimes hear from people asking whether spinrite is able to similarly repair and restore like an Android smartphone or other devices. And we tell such people that if their device allows itself to be placed into a mode where its storage is visible as a storage drive, then the chances are very good that as Bob found with his well used Kindle, spinrite can restore the device's proper operation and its prior performance.

[Leo Laporte]

Kind of amazing.

[Steve Gibson]

So very cool.

[Leo Laporte]

That's great. That even works on a Kindle. Alan, remind us, but before you go on, what did you do for the Speak and Spell?

[Steve Gibson]

I was involved in the lpc, the linear predictive coding of the speech. I was, I was involved at it.

[Leo Laporte]

The AI lab at Stanford had so early speech synthesis.

[Steve Gibson]

It was very early speech synthesis. Yes. Wow. Yeah.

[Leo Laporte]

Impressive.

[Steve Gibson]

It was fun. And back then, I want to say it has 4k bit of ROM.

[Leo Laporte]

That sounds about right.

[Steve Gibson]

It's about right.

[Leo Laporte]

And it's just like 1k and it's.

[Steve Gibson]

Yeah, 4.4kilobits of ROM. That's half a kid. And so for that thing to speak at all is crazy, you know, I mean it sounds awful. It sounds like a robot. I've got it back here. I'll stick it back. I'll stick some batteries in like spell. Spell relief, you know.

[Leo Laporte]

But you can understand it. I mean, it worked.

[Steve Gibson]

Oh yeah, it did work and it was astonishing at the time.

[Leo Laporte]

So it's a shame you didn't keep it up. You could be making hundreds of millions of dollars a year in AI research. Research. Right.

[Steve Gibson]

Well, this was done. I don't know if TI ended up with the patents on LPC linear predictive coding, but I think. But I know that Stanford produced some of the early work and research, so maybe they took it and refined to it. Yeah, as often happens.

[Leo Laporte]

Yep.

[Steve Gibson]

Alan Haig said. Hi Steve, love the podcast for decades now. And Spin. Right. The new version really helps with my TiVo drive, which is large. He said in a recent security. Now you mentioned that you no longer worry about AI since it has no intent. He said, I like the I want a lollipop scenario. But it seems to me that if AI has ingested all sci fi books, then it has many ideas of what a human might do when threatened. Could AI simply respond to a stimulus by using what it's learned? What it has learned shows could be a proper response. Couldn't it therefore replicate itself, disable electronic controls, or worse, without intent? Thanks for all you do for us all, Alan Haig in Indianapolis. And just a note. His note reminded me of my TiVos, and I know you had them too, Leo, which I still miss to this day. You know, that company for its day got so much right. You know, while we have vastly more options than we once did, it was once so nice having everything gathered in one place. Today it's necessary to go hunting around for shows among so many disparate services. But in any event, it's very cool that spinrite is still useful in keeping Alan's Tebow alive. And as he says, TiVo's drive being large means that before Spinrite 61, a full drive recover and refresh cycle would have taken quite some time, during which there would have been no media recording or playback on the TiVo. So with 6.1 being so much faster, that means much less downtime. And props to you, Alan, for keeping your TiVo going. I was forced to give mine up some time ago when I went digital and I wanted to play with all these other services. As for Alan, as for his notes about AI's possible negative reactions, I'm still 100% happy having settled upon the statement that mankind has not yet created an artificial intelligence. What we've been working toward for the past 100 years, although, you know, very rudimentarily in the beginning, amounts to increasingly good simulated intelligence. I really like the term simulated intelligence. I like it because it delineates itself from true intelligence, I think, in exactly the right way. And I believe that it helps us to disentangle ourselves from the very seductive struggle to understand what it is that we've most recently created. You know, we clever monkeys have managed to create an extremely convincing and compelling simulation of true human intelligence. But no matter how good that simulation may be, it's fundamentally different from the actual human intelligence that went into creating it. You know, a recording of an opera singer can be indistinguishable from the original singer, but the recording is not the opera singer. You know, a simulation is not the same as the real thing. Then, to your point, Alan, if an AI trained on sci fi, as certainly they will all have been at Least in part if they've been trained on Internet accessible material, because there's a lot of sci fi available on the Internet. If it were to be prompted with language that's threatening, and if it was not otherwise restrained from answering without filtering, I agree it would be likely to respond according to its training, which might be as we would expect a truly intelligent machine to respond because that's what it's simulating. But that would only be because what we have today are high fidelity simulations of truly intelligent machines. 40 years ago, 440 years ago. Edsger Dijkstra, the quite famous Dutch computer scientist and professor who's considered to be the father of structured computer programming, he created, he was the inventor the, the, the first conceiver of the notion of what we now call structured computer programming. He wrote an essay about the similar claims being made at the time of intelligent machines. And this was 40 years ago and, and previous, you know, so he, he was writing retrospectively. One of the things that he wrote in his takedown of this concept of machines being intelligent was so pithy that it stuck with me. He wrote that the question of whether computers can think is just as relevant and just as meaningful as the question of whether submarines can swim.

[Leo Laporte]

I love that.

[Steve Gibson]

Isn't that good? Wow.

[Leo Laporte]

Let me think about that a little bit. That's great.

[Steve Gibson]

Yeah. He wasn't. He said the question of whether computers can think is just as relevant and just as meaningful as the question of whether submarines can swim. So he wasn't a believer. And at least as regards what we have today, I think he is still just as correct as he clearly was 40 years ago. Today we may have far better submarines, but that does not make them fish.

[Leo Laporte]

Love it.

[Steve Gibson]

Eric Southwell said. Hello Stephen Leo, long time all caps his emphasis. Listener of the show. I eagerly await each episode. Getting to the point. I don't understand why this age verification problem is so intractably hard to solve. The US government has several databases that must include all of us for citizens. The Social Security Administration has all of our info for non citizen residents. The government has other databases that have unique numbers associated with people and importantly their birthdays. Can't we invent a secure process where we people somehow generate a hash or are provided a hash of just our name and birthday? Possibly we generate this hash on a government website that asks for other data to prove who we are. Later, when asked to prove our age to a different website, we provide the hash. The hash can be checked against the database of hashes for proof of age. The only Data transmitted is the hash data by design. The process from the age verifying service would be only yes to allow access or no to prevent access. I'm probably missing something obvious. It's just that it seems like we could use cryptography to provide data that's anonymous to a requester, but can be verified against a database that already exists in order to prove our age or identity. He says, Heck, maybe a QR code would do the trick. Or even a TOTP from an authenticator app or public private key pairs. Anxious to hear your thoughts, Eric. Okay, so there are two primary issues. The first is spoofing. As long as there have been age based restrictions on what someone can do or cannot do, there has been pressure to spoof one's age. The concept of the fake ID is so ubiquitous and deeply rooted right into our culture. Yeah, yeah. That it's not even a meme any longer. It's way beyond a meme. You know, there's no one who hasn't heard of a fake id. The, the primary classic reason for having and using a fake ID is so that its holder may fraudulently assert that they're older than they truly are. In the physical world, a higher quality fake ID will sport a photo of its underage holder. This makes contesting the ID when it's presented much more difficult. The other use case is the use of someone else's, you know, some other older person's actual id. In that case, the question is whether the photo on the ID is that of the person presenting it. In the first case, we have a falsified identifier for the person holding the id. And in the second case, we have a legitimate identifier for a different person. So the first and largest problem as we transition into the cyber realm is how to prevent the spoofing of anyone's age assertion. This is why I've consistently referred to the need for tight biometrics as a necessary component of any effective online age verification system. If some, if someone simply has a hash or a QR code or a public private key pair, nothing prevents any of those technologies, which are all inherently anonymous, from being shared with others. The time that would be required for an underground marketplace in fake online age assertions to be established would best be measured in microseconds. You know, I mean, it would, it would just be. It would instantly come into existence. Therefore, any technology that asserts someone's age must, you know, absolutely must somehow be tied to unspoofable biometric parameters that uniquely identify that person. This suggests that facial and Fingerprint recognition pretty much need to go hand in hand with any form of online age verification at the time of its assertion. And this of course presents a sticky wicket because not everyone has uniform access to some sort of biometric technology. But I said there were two important issues. The second one is no less important, and that's privacy. It will almost certainly be very important to people who wish to authenticate their age, for whatever reason there may be, that they not be individually identified as part of the requirement for doing that. This is where last week's zero knowledge proof business comes in. We need the ability to make a go, no go, you know, over 18 years of age or not, or maybe it's over 13, or maybe it's Apple's age ranges. In order to create, you know, fuzziness, we need to make that assertion and that assertion alone, without revealing anything else about ourselves. And this suggests that we need some sort of proxy we, you know, a proxy to which we biometrically authenticate to then make this assertion on our behalf. But we also don't want that proxy to obtain any information about the website to which we wish to authenticate. So we need to have a lot of blinding here. So, for example, the cryptographic tools we already have and already know well kind of provide us with a framework for a solution. For example, just off the cuff, some, some website that must authenticate our age before we're permitted to enter could present a large cryptographically unique random token. We've talked about many times before how trivial this is to generate. The site simply encrypts a counter which only ever counts up using a secret per site key. The output of that encrypted counter will be a pseudo random token that has never been seen before and will never be seen again. To this token, the site appends the age assertion that the site requires its visitor to validate. The user then needs to arrange to have that compound token signed by an age assertion provider. This could be anyone who participates in the system, like Apple or Google or Samsung, who have the necessary biometrics on their device, or anyone who's able to assert that they will somehow arrange to only ever sign an age assertion for someone whose age they have verified matches that assertion. So it can be, you know, broadly specified, but that, that whatever that agency is, their reputation is on the line. That when they sign this assertion, and note that the entity that's being presented this age assertion to sign knows nothing about the entity or website that generated the assertion. It's just a random token with an age assertion appended to it, so the user's privacy is preserved. The signed assertion is then returned to the user and from the user then to the website which verifies that the assertion is one, that it recently issued that, and that it has not yet been used since it must be single use, and that it matches the token that was issued for this user's current browser session so that that signed assertion from someone older can't be sent, given off to somebody younger to use. It's got to be the current browser session. The assertor's signature is verified against the root certificates in what would become, in this future environment, the industry's common age assertion root store. You know, in the same way that we have web browser root certificates, we would have age assertion root certificates in a, in a common store and the users then, you know, common storage and the users then admitted having passed these tests to the age controlled website. And so, you know, in, in this scenario there are a lot of manual processes with some additional thought it would be possible to automate and streamline this process using QR codes and so forth. So my point is this is a solvable problem. This is, this is not beyond us, but it needs to happen now. I'm, it's from. To me, it's extremely annoying that the U. S. Supreme Court ruled that no one's first amendment rights to protected free speech would be abridged by the imposition of this quite onerous requirement that is, you know, age verification. You know, as we all know at present, the industry has no means whatsoever for asserting anyone's age without sacrificing all of their privacy and their individual identity. So it's like, you know, it's very much like the UK exactly like the UK saying, you know, you must give us access to everyone's messages or any, no, sorry, to anyone's messages we ask for while absolutely preserving everyone's privacy. You can't have it both ways. And so here's the Supreme Court just saying, yeah, everyone must be able to assert their age. But no one's going to do that because it's going to be a complete loss of privacy and we have no, no mechanisms in place for this. So, you know, this imposition of age restriction significantly changes the nature of the Internet. Some of our listeners have forwarded links to me since I began talking about this. More to commentary written by authors of websites containing, for example, salacious adult content that's far more tame than the legislation's initial targets are aimed at. Yet that adult content falls well within the Very broad legislation's scope. So the point has been made that this is only the initial foray and that the underlying goal is to force the removal from the Internet of any content that a minority of the U. S. Public may find unacceptable. You know, the Internet we have tomorrow may look much different from the one we have today. In some ways it'll be better, but unfortunately the control that is now beginning to be asserted can always be misused. So. I don't know, Leo.

[Leo Laporte]

Yeah, well, we just watch and this is the place to watch.

[Steve Gibson]

We are gonna, we're gonna make that technology happen right here.

[Leo Laporte]

Yeah. I think there may also be legal reasons you can't use the Social Security Administration or the IRS databases to verify age.

[Steve Gibson]

Like actually the law says you can't use a Social Security number. Yeah.

[Leo Laporte]

For identification. And I don't. I think that there are lots of reasons why that data, despite the fact that DOGE is actually now trying to unify it all that data should be protected from widespread use for other purposes.

[Steve Gibson]

I think we clearly need a privacy preserving, you know, some sort of age assertion system that everyone understands isn't revealing anything about them. But whether or not they are a.

[Leo Laporte]

Certain age, I'm sure somebody's got to be working on that. I.

[Steve Gibson]

Stina referred to something called WW Wallet, which she said was an open source effort and there are some. The EU has some sort of a wallet technology. I, I don't yet know, you know, what the details are. And you know, it's got to be widespread, it's got to be in our smartphones. And I just. The other thing that bugs me, Leo, is that, is that I can't see how this cannot, how this can possibly be free. So what we're saying is that that's not okay either. Exactly. We are having to deed. Democratize the Internet right now. You know, if there, you don't, you're not charged for access other than putting up with ads and tracking. But I can't see how we're going to be able to verify age without some technology that involves biometrics. And that can't, I don't see how we make that free.

[Leo Laporte]

I think the best, honestly the best we can do is to, is to put it in the hands of the parents. I know that's not a perfect solution. You can't.

[Steve Gibson]

Well, that only, that only solves the problem with kids. How do adults who want to prove their age do?

[Leo Laporte]

Well then you don't have to because you're presumed if you don't have a parent blocking you that you're an adult.

[Steve Gibson]

That doesn't work for what the, what the US Constitution just said. The us. I mean, the US Supreme Court just said that sites can require positive confirmation that some actual age over 18.

[Leo Laporte]

Right.

[Steve Gibson]

And, and so it's not just so. So if so it would be a, it can't just be a, a device saying that I'm old enough because, because there's no proof of that being true. You know, 18, you know, 17 year olds could, could have a device that says that. So it's a mess. But it's time for a break and then we're going to talk about Cloudflare.

[Leo Laporte]

Okay, you're watching security now. You see, we deal with the intractable issues of the day and we attempt to solve them with logic and, and thought as opposed to emotion. And that's what Steve's so good at. We're glad you watch. We encourage you, if you are a viewer, to support the show by joining the club. 10 bucks a month gets you ad free versions of this show. In fact, all the shows we do access to the Club Twit, Discord. A great place to chat about this and everything that's going on. I mean, the Club Twit is. Discord is kind of a social network that goes 24 7. It's my favorite social network. You also get special shows that we don't do anywhere else if you're interested, and I hope you are because it really makes a difference to what we can do as a, as a network. Twit tv, Club Twit. And thank you in advance for your support. We really appreciate it. We've got a lot of great members. I just, I'm thrilled about the club. It's doing very well. But it would do better if you were a member. It really would. Let's map out this week's amazing destinations and travel tips.

[Will]

Honestly, Will, I didn't plan any trips, but I did switch to T Mobile with their new Family Freedom offer.

[Leo Laporte]

That's not the itinerary we're following.

[Will]

Well, I'm departing from AT&T and embarking on a new journey with T Mobile. They paid off my family's four phones up to $3200 and gave us four new phones on the house.

[Leo Laporte]

Bon voyage.

[T-Mobile Announcer]

Introducing Family Freedom. Our lowest cost will switch our biggest family savings all on America's largest 5G network. Visit your local T Mobile location or learn more@t mobile.com FamilyFreedom up to $800 per line via virtual prepaid card. Typically takes 15 days free phones via 24 monthly bill credits with finance agreement eg Apple iPhone 16128 gigabyte 82999 Eligible trade in eg iPhone 11 Pro for well qualified credits end and balance due if you pay off early or cancel Contact T Mobile.

[Steve Gibson]

From unsolved mysteries to unexplained phenomena, from comedy goal to relationship fails, Amazon Music's got the most ad free top podcast included with prime. Because the only thing that should interrupt your listening is, well, nothing. Download the Amazon Music app today.

[Will]

You can make a difference in someone's life, including your own with a job in home care. These jobs offer flexible schedules, health care, retirement options and free training. They also provide paid time off and opportunities for overtime. Visit oregonhomecarejobs.com to learn more and apply. That's oregonhomecarejobs.com all right, let's talk about.

[Leo Laporte]

Can you call it quad one. 1.1?

[Steve Gibson]

Yeah, actually that would be a lot easier than. Yeah, that would be a lot easier than saying 1.1.1.1 every time. So I'm going to say that during the podcast. Quad one.

[Leo Laporte]

Okay.

[Steve Gibson]

Because I've written 1.1.1.1 and I got tired of just writing.

[Leo Laporte]

It's a lot. It is dotted quads were not designed to be typed.

[Steve Gibson]

So I have not mentioned anything about my discoveries resulting from my pretty much incessant use of the new and still developing GRC DNS benchmark.

[Leo Laporte]

I'm excited about this.

[Steve Gibson]

It's really current, turning out very cool. Yay. But I'm just, I'm, I've just added a new feature and it's like, oh gosh, where's the last one I'm putting. But I thought it would be very cool to allow its users to put to enter any domain name they want and then check it against all the DNS providers in the list to see whether they are filtering it or not to, to be a DNS filter checker in addition to just being a performance checker. So it's getting, it's, it's broadening its scope a little bit, but in ways that I think are useful for the future. So, you know, and I don't want to go back to it again, so I'm putting, you know, everything in that I can think of that would be useful. So what I suspect most of the benchmarks users are, are going to discover is that if you didn't have something like the benchmark to more carefully customize and personalize or confirm your own choice of optimal DNS resolvers, you probably could not go very wrong choosing any of Cloudflare's DNS solutions. Although they're not alone among the benchmarks top rated resolvers, they're always near the top. Cloudflare is, and I've been quite impressed with what I've seen. I'll have a lot more to say about that before long. I'm mentioning this today because exactly one week ago, as I mentioned at the top of the show, on July 14th, while we were recording this podcast from just before 3pm to just before 4pm so right now as it's 3:50 so one week ago at this time 1.1.1.1 I'll say it one time Quad One was gone. It was not resolving, it was off the air, which is, you know, earth shaking really. Because this this resolver is so popular they suffered Cloudflare suffered a significant outage which caused their wildly popular primary DNS resolver, that Quad One ip, to disappear from the Internet for an hour. The details surrounding this event are extremely interesting and I thought everyone would enjoy learning about not only what happened, but also why and how. So before I start by sharing the introduction of their report, I want to note that this is precisely why standard best practice on the Internet has always been to configure a pair, at least a pair of DNS resolvers for use by every connection to the Internet. Stuff happens, as they say. So anyone whose Internet connection was configured to use both of Cloudflare's IPs Quad 1 and its secondary backup of 1.0.0.1, assuming that 1.0.0.1 did not also go offline, and I was never able to confirm that either way, I'm not sure that they're not referring to both as 1.1.1.1. So it might actually be it might make more sense to use a different provider for if not a secondary than a a tertiary DNS. But in any event the concept is to have two different DNS resolvers and if you had that and assuming that quad one went off the offline but 1.0.0.1 did not, then users would have only noticed a brief stutter when when Quad one stopped responding operating systems, all of them that their their TCP IP stacks that do this DNS resolution, they will first reissue their UDP DNS queries under the assumption that the UDP packet that went out and tried to come back may have been dropped either to or from that remote resolver. Then once the primary resolver has failed to respond to a couple of retries. All DNS resolvers that are configured on that Internet interface will simultaneously be queried in parallel and the OS will then switch to using the the first one to reply. So a nearly transparent switchover from quad one to 1001 would have occurred for many people during that hour long outage. Just you wouldn't maybe have noticed anything if assuming that 1001 had stayed up. And one last point, lest anyone worry that their that their LANDS network border router may only be assigning a single DNS IP which is aimed at itself to their PCs inside the LAN. This is a common configuration and it should not be any cause for concern in these scenarios. The LANS router is serving as the proxy for the public facing DNS resolvers and is using DHCP Dynamic Host Configuration Protocol to configure the client machines on its Lantern to ask it for any of their DNS resolution needs and then it will in turn forward those DNS queries to one or more of its configured public resolvers which are in turn often configured and provided by the connections ISP using also DHCP on the WAN side interface okay, so what happened at Cloudflare to cause a massive hour long worldwide outage of their flagship DNS resolver? Here's what they shared. They wrote on July 14, 2025, Cloudflare made a change to our service topologies that caused an outage for 1.1.1.1. I can't help myself saying it. Quad One on the edge resulting in downtime for 62 minutes for customers using the Quad One public DNS resolver as well as intermittent degradation of service for Gateway DNS. Cloudflare's Quad1 resolver service became unavailable to the Internet starting at 21:52 UTC and ending at 22:54 UTC. The majority of 1111 users globally were affected. For many users, not being able to resolve names using 1111 Resolver meant that basically all Internet services were unavailable. This outage can be observed on Cloud Flare Radar okay, now I'm going to pause here because this Radar page of theirs is very cool. I have its link in the show notes and I've also made it this week's GRC shortcut. So you can just go to GRC SC 1035 GRC SC today's episode number 1035 or click the link in the show notes and that just bounces you to the same place. Anyone who is interested in DNS at scale will find this page very interesting. For example, the second chart shows the overall usage ratios of the four DNS protocols for their Quad 1 resolver.

[Leo Laporte]

I wouldn't have thought this at all.

[Steve Gibson]

I know. So, wow. Traditional DNS over UDP currently commands an 86% share. In a very distant second place is DOT at 7.1%, then DOH at 4.7 and played unencrypted TCP at 2.2. Huh. Now, although modern browsers have settled upon using DOH for their use of privacy enforcing DNS, when Android devices are configured to use private DNS with cloud flares, that's dot or actually any private DNS. The various private DNS's are that Android devices can be configured for are dot by default and dot is often preferred by IoT devices and enterprises. So that's why it's in second place. Although. Wow. A very distant second place, you know, at 7.1%. And of course, you know, the reason is DNS has always been udp, so it still holds, it's, you know, a grip on 86% of all DNS resolutions. Another interesting data point is that Cloudflare's Quad 1 resolver receives 62.6%, so just just shy of 2/3 of its requests for IPv4 addresses, whereas queries for IPv6 addresses make up 18.8% or so. So IPv6 requests is nearly 1/5 of the total, whereas IPv4 is 2/3. So yes, it's clear that IPv4 still rules, although, you know, less than I.

[Leo Laporte]

Would have thought, to be honest.

[Steve Gibson]

Yeah, yeah, yeah, exactly. 20, you know, for, for 20. Nearly 20. 18.8 to be IPv6. That's still pretty good.

[Leo Laporte]

Yeah, I mean, obviously people are using 11. 11 are more sophisticated than a normal user. Right?

[Steve Gibson]

Yeah.

[Leo Laporte]

They have somebody fancy in the house.

[Steve Gibson]

They'Ve deliberately chosen that.

[Leo Laporte]

Right.

[Steve Gibson]

Because it's not their ISPs. DNS.

[Leo Laporte]

I would suspect fewer than 1% of all Internet users use a custom DNS.

[Steve Gibson]

Yes, yeah, it's just, look, it works and it's, you know, it's going to go to their isp, who's rubbing their hands together because they're getting all of the DNS. So I'll note that the DNS benchmark tends to favor resolvers having IPv6 addresses, meaning that it consist the GRCs. DNS benchmark, which now supports all of those protocols, IPv4, IPv6 do and do consistently finds that that resolvers with IPv6 addresses responds slightly faster than resolvers addressed with IPv4. And Cloudflare does have a similar pair of IPv6 resolvers, but my God, Leo, the IP is just from hell. I mean, it's like, it's like, well, you know, all the other long sixes. Yeah, yeah. And so it's not fun to say or fun to type, but once you do it, you end up with slightly faster DNS. So anyway, lots of interesting stuff on that page. I commend it to anybody who's interested. So let's continue with Cloudflare's explanation of who tripped over what cord at headquarters. They wrote the outage occurred because of a misconfiguration of legacy systems used to maintain the infrastructure that advertises. Okay, and this is weird jargon that, that BGP uses, so I'll explain this in a minute. That advertises Cloudflare's IP addresses to the Internet. This was a global outage. During the outage, Cloudflare's 1.1.1.1 resolver was unavailable worldwide.

[Leo Laporte]

Wow.

[Steve Gibson]

We're very. Yeah, I know, it's just, it's breathtaking. We're very sorry for this outage they wrote, period. The root cause was an internal configuration error and not the result of an attack or a BGP hijack. In this blog we're going to talk about what the failure was, why it occurred, and what we're doing to make sure this doesn't happen again. They wrote. Cloudflare introduced the Quad One public DNS resolver service in 2018. So that's interesting to know. It is seven years ago. Since the announcement, Quad One has become one of the most popular DNS resolver IP addresses and it is free for anyone to use. And yeah, like why wouldn't you use it? I mean it is often faster. Actually, I wonder if it's never not faster, which would be to say, as always, faster than the isp. I think it's always faster than my Cox, you know, automatically assigned DHCP IP or DNS resolution, which is astonishing, but we'll talk about why in a minute. They wrote, almost all of Cloudflare services are made available to the Internet using a routing method known as Anycast, a well known technique intended to allow traffic for popular services to be served as in many different location, or it should say served from many different locations across the Internet. Increasing capacity and performance. This is the best way to ensure we can globally manage our traffic. But also means that problems with the advertisement of this address space can result in a global outage. Okay, so let's talk about. Let me take a break here and Talk about anycast for a second. Several weeks ago we mentioned that the European Union had introduced a set of its own DNS resolution services for its EU member citizens. I immediately added all of their DNS IP, dot and doh addresses to GRC's default list of resolvers in for for the benchmark. And I remembered mentioning on the podcast that I was quite a bit put off by their sluggish performance. In retrospect, this was to be expected since the benchmark was actually communicating with DNS resolvers operated by Whalebone and located in the Czech Republic. And while that might be right around the corner for users in the eu, it's on the far side of undersea cables and many router hops from my location in Southern California. I confirmed subsequently with many of our EU located DNS benchmark pre release testers that the same DNS for EU resolvers operate quite acceptably well for anyone who's located near them. In other words, for those DNS for EU resolvers, their actual real world performance will be a direct function of how far away the client is from the location of those physical servers whose IP addresses the client is accessing. These resolvers have so called unicast IP addresses where traffic addressed to those addresses will be routed across the Internet to wherever it is they're located wherever their servers are. And this is completely fine for EU citizens since those servers will be close by and the EU certainly doesn't wish to expend their resources arranging to make their DNS for EU fast. For me in the United States, that's not a priority for them. Okay, so what's different about Cloudflare and their Quad1 IP? That. That. That 1.1.1.1 Cloudflare IP is an Anycast address where the IP does not refer to any specific physical resolver resolver hardware. So any traffic addressed to that IP is not routed to some resolver located at a specific location. Instead, any CAST addresses will automatically route to the closest Cloudflare data center. This means that whereas the performance of the DNS for EU IPs is determined by the client's location and their distance from the eu, Cloudflare, being a major global network provider, will have a data center that's close to everyone, and that single ubiquitous Quad One IP will automatically cause any client's DNS lookup traffic to be routed to that closest data center for its resolution. It's an extremely slick system. I mean, it's the way CDNs operate and it explains how Cloudflare is able to offer Their super high performance DNS services from a single universal ip no matter where its clients may be located. You know this is the, you know the whole idea of a CDN having a, an the an edge appearance meaning you know near you, meaning an edge of their network is very few router hops away from your locate from where you're located. Okay, so back to Cloudflare. They wrote Cloudflare announces these anycast routes to the Internet in order for traffic to those addresses to be delivered to a Cloudflare data center providing services from many different places. Most Cloudflare services are provided globally like the 1.1.1.1 public DNS resolver, but a subset of services are specifically constrained to particular regions. These services are part of our data localization suite which allows customers to configure Cloudflare in a variety of ways to meet their compliance needs across different countries and regions. One of the ways in which Cloudflare manages these different requirements is to make sure the right services IP addresses are Internet reachable only where they need to be so your traffic is handled correctly worldwide. A particular service has a matching service topology that is traffic from for a service should only be routed to a particular set of locations. Now Leo, they just said that they're talking about data local localization suite and services only being available within server specific locations. Which sounds suspiciously like this large collection of domains that dropped off the net for the UK at exactly the same time of this outage.

[Leo Laporte]

Oh, very clever.

[Steve Gibson]

Isn't that interesting?

[Leo Laporte]

All right, we're going to pause right here for just a bit. Stay tuned. More security now. Still to come, today's show is brought to you by Progressive Insurance. Fiscally responsible financial geniuses, monetary magicians. These are things people say about drivers who switch their car insurance to Progressive and save hundreds. Visit progressive.com to see if you could save Progressive Casualty Insurance Company and affiliates. Potential savings will vary. Not available in all states or situations. Let's map out this week's amazing destinations and travel tips.

[Will]

Honestly Will, I didn't plan any trips, but I did switch to T Mobile with their new Family Freedom offer.

[Leo Laporte]

That's not the itinerary we're following.

[Will]

Well, I'm departing from AT&T and embarking on a new journey with T Mobile. They paid off my family's four phones up to $3200 and gave us four new phones. On the house.

[Leo Laporte]

Bon voyage.

[T-Mobile Announcer]

Introducing Family Freedom. Our lowest cost will switch our biggest family savings all on America's largest 5G network. Visit your local T Mobile location or learn more@t mobile.com familyfreedom up to $800 per line via virtual prepaid card typically takes 15 days. Free phones via 24 monthly bill credits with finance agreement eg Apple iPhone 16128 gigabyte 82999 eligible trade in eg iPhone 11 Pro for well qualified credits and balance due if you pay off early or cancel contact T Mobile this episode.

[Leo Laporte]

Brought to you by Red Canary when cybersecurity threats hit fast, you need an MDR partner that moves faster. Red Canary delivers 24.7expert MDR support, total visibility and actionable insights. Plus it helps you detect four times more threats so you can stay ahead without burning out. Red Canary clears the noise and has your back every hour, every incident. Get the backup you deserve. Visit redcanary.com difference to learn more. All right, back to the show.

[Steve Gibson]

Steve so they wrote on June 6 during a release to prepare a Surface topology for a future DLS service, a configuration error was introduced. Get this the prefixes associated with the Quad One Resolver service were inadvertently included alongside the prefixes that were intended for the new DLS service. Ok, so just to be clear that that fundamental configuration error which lumped the universal availability of the Quad 1 DNS IP in with some others, occurred back on June 6 when they were preparing, not in July. So it was more than a month ago, more than a month old when it happened, they explain. This configuration error, they wrote, sat dormant in the production network as the new DLS service was not yet in use, but it set the stage for the outage on July 14. Since there was no immediate change to the production network, there was no end user impact, and because there was no impact, no alerts were fired. Their report then lays out a detailed minute by minute and hour by hour timeline of what they call the event at 2148 UTC just before 3pm during last week's podcast recording, you know the you know what started to hit the fan. They detailed this they said a configuration change was made for the DLS service. The change attached a test location to the non production service. This location itself was not live, but the change triggered a refresh of network configuration globally, meaning a BGP rerouting of traffic. And I'll explain more about that in a second. Everything they they they said Due to the earlier configuration error linking the Quad1resolver IP address to our non production service, the Quad1IP was inadvertently included when we changed how the non production service was set up, the Quad 1 resolver prefixes started to be withdrawn from production Cloudflare data centers globally. Okay, so as I said, everything we're talking about here is, is bgp, the border Gateway protocol, which we've covered a number of times in the past. Generally with bgp, when something goes very wrong with the Internet due to its misconfiguration, that's what's going on. You know, such as, you know, a mistake attempting to route all of the Internet's global traffic through a pawn shop in Lower Slovia. You know, that never turns out well for anyone. So something similar happened again and with a similar outcome. Internet traffic is great and it works incredibly well right up until it utterly fails, and then it generally fails big. So at 2152 they wrote DNS traffic to quad one resolver service begins to drop globally. At 2201, Internet service health alerts begin to fire for the 1.1.1 resolver and a formal incident event is declared. 2240, a fix is deployed, a revert was initiated to restore the previous configuration to accelerate full restoration of service. A manually triggered action is validated in testing locations before being executed. At 22:54, the impact ends, resolver alerts are cleared and DNS traffic on resolver prefixes return to normal levels. Okay, so what was the impact of this? There are some interesting details there too. They write when the impact started, we observed an immediate and significant drop in queries over UDP, TCP and DNS over TLS. And they wrote most users have 1.1.1.11.0.0.1. And then they list their two IPv6 IPS, which are 2606, 4700, 4700 colon colon 11 or same thing and then 1010 configured as their DNS server. They said it's worth noting that DoH, the DNS over HTTPs traffic remained relatively stable as most DOH users use the domain cloudflare hyphen DNS.com configured manually or through their browser to access the public DNS resolver rather than by IP address. DoH remained available and traffic was mostly unaffected. As Cloudflare DNS.com uses a different set of IP addresses, some DNS traffic over UDP that was also used different IP addresses remained mostly unaffected as well, they said, as the corresponding prefixes, meaning BGP routing prefixes were withdrawn. No traffic sent to those addresses could reach Cloudflare. As we we can see this, they said in the timeline for the BGP announcements. And there is a. It's lower down on, on that same radar page I talked to before. You see a spike in traffic where the withdrawals happen and then an hour goes by and another spike when the announcements when, when the proper prefixes are being re. Announced. So the, the, the second spike is when they have realized how to fix what has gone wrong and then, and then apply, apply that and the announcement of the update to the routers spreads out across the Internet. One last bit of interesting charting that they provide I thought was very cool. It's shown as a, as a green chart down at the bottom of the radar. They said. When looking at the query rate for of the withdrawn IPs it can be observed that almost no traffic arrives during the impact window when the initial fix was applied. At 22:20 UTC, a large spike in traffic can be seen before it drops off again. This spike is due to clients retrying their queries. When we started announcing the withdrawn prefixes again, queries were able to reach Cloudflare once more. It took until 2020, 22:54 UTC before routing was restored in all locations and traffic returned to mostly normal levels. So it's very cool that that chart shows the 90 minutes before the event. Everything's just, you know, puttering along more or less straight line. Then it just utterly disappears. Bang. It's like a sharp edge drops straight down to zero. Which is what we would expect once the entire Internet has essentially forgotten what to do with that ip. That's what this means, is that for the Internet, all the routers on the Internet have just, they've, they have no idea what to do with those IPs. Then at 22:20, the traffic just as suddenly skyrockets to about six or seven times its normal level. And as they wrote, DNS clients that were at that moment just discovering the outage and were, were retrying would have been frantically sending DNS packets out, retrying their queries, you know, basically creating an artificial tsunami, which you know, can be seen at the Cloudflare resolvers once routing had been restored. Their post mortem posting then digs deeper into how and why this happened. I'll share one paragraph of it and see if this doesn't sound hauntingly familiar to what we heard Crowdstrike explain almost exactly one year ago last July, after they caused the crash of 8.5 million Windows machines. Cloudflare wrote this is, you know, just, just last week, Cloudflare wrote the way Cloudflare manages service topologies has been refined over time and currently consists of a combination of a legacy and a strategic system that are synchronized Cloudflare's IP ranges are currently bound and configured across these systems that dictate where an IP range should be announced in terms of data center location on the Edge network. The legacy approach of hard coding explicit lists of data center locations and attaching them to particular prefixes has proved error prone since, for example, bringing a new data center online requires many different lists to be updated and synced consistently. Okay, and here it comes, they wrote. This model also has a significant flaw in that updates to the configuration do not follow a progressive deployment methodology does not progressive Even though this release was peer reviewed multiple engineers by multiple engineers, the change did not go through a series of canary deployments before reaching every Cloudflare data center. In other words, just as with CrowdStrike, there was what turned out to be too much confidence placed in their automation, so deployment was all at once and not incremental or tested in, you know, basically in place before it was let loose upon the entire planet. And as they say, lessons learned. After sharing a bunch more detail, including how the inadvertent withdrawal of the Quad 1 routing revealed an underlying but inconsequential BGP hijack originating from Tata Communications in India, they conclude writing Cloudflare's 1.1.1.1 DNS resolver service fell victim to an internal configuration error. We're sorry for the disruption this incident caused for our customers. We are actively making these improvements to ensure improved stability moving forward and to prevent this problem from happening again. And after rereading all this LEO and, and seeing that they, they talk about all four of those IPS together, my guess is that they're always referring to them collectively as their 1.1.1.1 resolver, but that all four of those probably dropped off the Internet because they would have all four been served by the same data centers, all which stopped receiving their, you know, incoming packets. So my guess is that if somebody as most people would have had 1, you know, 1111 and 1, 001 configured as their primary and secondary DNS, I'll bet you for an hour they had no Internet access appreciably, I mean no effective ability to look up the DNS addresses, to look up the ips of their domains. So which explains the mea culpa there because it's like, yikes, yes, that would have been a problem. So they are already moving forward toward a better and less error prone system to support their future growth. If nothing else, this mishap, much as it showed CrowdStrike a year ago, showed them the value of the planning that they have been undertaking and, and deploying and that, you know, they're making a necessary and important investment. I've got all the links to the original report and the cloud radar graphs at the end of the show. Notes for anyone who's interested. And boy, if you were wondering if you were, as I imagine our listeners probably were. Yeah, 1111 and 1001. Now you know what happened just shows.

[Leo Laporte]

How dependent on a DNS resolver we are.

[Steve Gibson]

I mean completely. I mean it is so crucial to the operation of, of all of the services that we now just take for granted on the Internet.

[Leo Laporte]

So where do we stand on the DNS benchmark Pro?

[Steve Gibson]

I'm. That last feature I mentioned is finished. The ability to, to do a large, huge wholesale analysis of filtering against domains. The bench.

[Leo Laporte]

That's really useful. I'm glad you're adding that. I mean, I. Adding features is a, is a way to slow it down. But that's a good one. I think that's a very useful.

[Steve Gibson]

I, I just think it, it really does make sense to be able to, to quickly see that with any domain name you want. So you just, you just put test domains in and it'll.

[Leo Laporte]

What I can't see is very important.

[Steve Gibson]

Yeah. It'll show you what you know and confirm that your resolver is, is filtering what you would hope it would be. Yeah, because Good point.

[Leo Laporte]

Yeah.

[Steve Gibson]

So anyway, and that's why 1.1.1.2 and 1.1.1.3 are now there because they are filtering DNS and you can actually see them in real time doing that. So anyway, so that's done. I got a little sidelined on throttling because it turns out that this thing is so busy, it's got so much juggling at the same time that throttling with the number of outstanding queries is tricky because the. I'm also checking like the DNS for EU resolvers and from where I am, they're very slow, so I don't want to. So that means that suddenly a lot of queries are outstanding and it tends to stall the benchmark. So I'm just in the process. When I, when I stopped working on it Saturday night to begin working on the podcast Sunday morning, I was in the process of, of coming up with a system which will age the queries that are outstanding and only throttle if they are younger than a cutoff so that I won't get penalized for resolvers that are taking much longer to reply. Anyway, the end user sees none of that they just go, well, look, it works. But anyway, I'm very close to being done. Windows 11 allows the OS itself to be configured to use DoH, so I need to do a little special handling of that. And then I need to spend some time with the command line features because I'm sure they're badly broken. But anyway, all the heavy lifting is done. It supports all the protocol. You know, people test. Every time I do a release, people write back and go, well, this just works. And it's been working for like the last 12 releases. So it's like, okay, I'm sure I'll break something. So anyway, we're getting close.

[Leo Laporte]

Having fun. That's the most important part.

[Steve Gibson]

Have fun. And I'm going to create a next generation very useful benchmark.

[Leo Laporte]

Nice. Well, here's a way you can keep up on its status and find out. The minute it's released is that's to going go to GRC.com email that does two things. One, you can submit your email address so that Steve can whitelist you so you can send him comments, suggestions, pictures for the picture of the week, that kind of thing. But you'll see two unchecked checkboxes below it for the two different newsletters Steve offers the weekly newsletter, which is the show notes from this show, and the very infrequent emailing about new products, things like that. If you sign up for both of those, then you'll be alerted the minute the DNS benchmark comes out. Plus you'll get the picture of the week a day early and you can laugh along with Steve. That's just one of the many things you could do. At GRC.com, he is of course his bread and butter. He's the creator of spinrite, the world's finest mass storage, maintenance, recovery and performance enhancing utility. Even works on your Kindle, which I never would have thought of, but it makes sense. Anything that's storage, right?

[Steve Gibson]

Yep.

[Leo Laporte]

If you don't have a copy of spin right, get 6.1 right now. If you do, make sure you're upgraded. Upgrades are free for everybody who's ever bought a version of [e-mail] he also has copies of the podcast there all he's got unique copies. We don't do any of the forms. He does now. He has a 16 kilobit audio version for the bandwidth impaired, a 64 kilobit for those of you who just want the audio and a good quality without a lot of fuss. He also has show notes which you can if you don't subscribe to the newsletter, download from there. That's great to read along while you're listening. He also has transcripts written by an actual human, Elaine Ferris. So in a couple of days after the show comes out, you'll be able to get that and that you can read along while you listen. But it also makes a great way to search for the stuff you want. All of that is at grc.com@TWIT TV SN we have audio, 128 kilobit audio and video of the show. You could find that there. Download it at your leisure. You can also get it on YouTube. There's a YouTube channel dedicated to it that we keep for a very special reason. It's a great way to share clips of the show because everybody has access. It's kind of the lingua franca. It's the easiest way to share video. So if you see something on Security now and you say, oh I gotta tell somebody about this, use the YouTube channel. It's a great way to share it. Tell people about the show. Easiest way to get the show, probably subscribe in your favorite podcast client. That way you don't even have to think about it. You'll get it automatically as soon as the show's done. Audio or video, again, whatever you prefer. Pocket Cast, you know, Apple's podcasts, there's all sorts of places to get it. If you do subscribe to one of those, please leave us a nice 5 star review to let the world know how great security now is. It helps us to spread the word and that's really, I think now that we've been doing this for almost 20 years. Be 20 years next month.

[Steve Gibson]

Yep.

[Leo Laporte]

The job one now is just to let the world know we've been doing this. We will continue to do it as long as Steve is up for it. And it's a really valuable resource. I think you'd agree or you wouldn't be here. You can watch us live. You don't have to watch after the fact. You can watch us do it live. We stream the show right after Mac break weekly. That's 1:30 Pacific, 4:30 Eastern, 20:30 UTC. Roughly. We're not a TV channel. You're watching us produce the show. So roughly those times, the live streams. Well if it's, if you're in the club, it's of course on the club Discord. And I apologize for the scratchiness of the live stream at the beginning. You know, sometimes that happens. We fix it as quickly as we can. There's also YouTube, Twitch, TikTok tock, Facebook, LinkedIn, X.com and Kick8 different places. You can watch us live every Tuesday afternoon. I hope you will come by and watch and chat with us, but do download the show also so that you know everybody knows we've got lots of security now.