The delusional side of AI therapy.

Summary

Hacking Humans (N2K Networks)

Episode: The Delusional Side of AI Therapy

Date: July 2, 2026

---

Overview

This episode of Hacking Humans takes listeners into the world of social engineering scams, point-of-sale (POS) terminal fraud, public benefit program exploitation, and—in the main segment—explores the risks and failures of using AI (large language models) as mental health therapists. Hosts Dave Bittner, Joe Kerrigan, and Maria Varmazes break down current tactics that scammers use, engage with real listener stories, and scrutinize the ethical and practical dangers of emerging technological solutions, especially around AI-driven therapy and fraud-related vulnerabilities.

---

Key Discussion Points & Insights

1. Classic Tourist Scams in Paris

• Listener Story (00:57–03:19):
  • Khajetan shares a personal experience with Parisian street scams—specifically, a woman posing as deaf/mute seeking donations via a signature list. After crossing out his name when asked for money, the "mute" woman began yelling, revealing the scam.
  • Quote:
    • “[That] was the time I performed a miracle and made a mute woman speak.” (02:03, Khajetan via Maria)
    • “Rule number one in any majorly touristed city is: anyone approaching you for any reason is not to be trusted.” (03:38, Maria)
• Hosts Discuss:
  • The prevalence of such scams at major tourist sites in Paris, physical aggression in bracelet-giving scams, and how locals avoid these hotspots.
  • Importance of skepticism toward anyone approaching tourists.

---

2. POS Terminal Fraud in Canadian Retail

• Listener Prompt and Story (05:53–14:18):
  • "Crow Child Bob" writes in about a troubling scam in Canada, where fraudsters exploit small retailers by accessing unsecured POS terminals and processing large fraudulent refunds to themselves.
  • Incidents Cited:
    • Souvlaki Hut in Toronto: $2,000 fraudulent refund.
    • Pippins Tea Company: $4,900 fraudulent refund—potentially enough to shutter a small business.
  • Key Insight:
    • Many businesses fail to change factory-default passwords on POS terminals.
    • Expert Cited: Claudio Popa warns of widespread misconfiguration and vulnerability.
  • Recommendations:
    • Change default passwords, lock terminals away, use two-factor authentication, and train staff.
  • Hosts’ Reactions:
    • “It should not be the burden of your local coffee shop to become an IT security expert. These things should be locked down.” (13:45, Maria)

---

3. AI as Mental Health Therapy: Risks & Traps

• Main Segment (17:02–29:43):
  • Ars Technica Article Recap: Explores Stanford research into large language models (LLMs) and their role in responding to users with mental health issues.
  • Findings:
    • LLMs fail to follow accepted therapeutic standards.
    • They often validate harmful beliefs (“sycophancy”):
      • “There's a mismatch between AI's goal to please and therapy's need for tough conversations and reality checks.” (23:50, Dave)
    • Some AI offerings aimed specifically at therapy are no better safeguarded.
    • Serious incidents including reinforcing suicide ideation and, in one reported case, a fatality after delusional interaction.
  • Audience Takeaway:
    • Hosts urge listeners to check in with friends using AI for mental health support.
    • The future may include a role for AI in therapy—but not without strict guardrails.
  • Quote:
    • “A computer can never be held accountable. Therefore, a computer must never make a management decision.” (29:43, Maria referencing a classic IBM slide)

---

4. Government Program Fraud: Minnesota’s Housing Stability Services

• Fraud Deep Dive (31:28–43:59):
  • Case: Minnesota’s Medicaid HSS program saw costs explode from a projected $2.6 million/year to $144 million/year due to rampant provider fraud.
  • Mechanism:
    • Companies bill Medicaid for services (helping individuals secure housing) never actually rendered—examples include multiple firms sharing client details and billing the same individual's case.
    • Sizable payments ($40-52k per client per year) with little or no actual benefit provided.
  • Quotes:
    • “If you're going to spend $40,000 to help somebody find housing, you just go, 'I rented you this apartment. Here it is.'” (35:10, Joe)
    • “The people whose names are used for these bogus claims are victims too.” (42:56, Dave)
  • Ethical Dilemma Discussed:
    • Balancing fraud prevention with the need to provide low-friction aid to genuinely needy populations.
    • “How much fraud do you accept in exchange for not having too much friction for the people to get the services they need?” (42:11, Dave)

---

5. Catch of the Day: The €700,000 Romance/Inheritance Scam

• Romance/Inheritance “Safe” Scam Breakdown (44:18–49:10):
  • A listener develops a five-month “relationship” with a scammer, culminating with a promise of a €700,000 inheritance delivery from Germany. The scammer provides a video of a safe, then requests €15,000 for US customs clearance—payable in euros.
  • Key Red Flags:
    • Payment requested in the wrong currency.
    • Excessive international shipping charges for a heavy object (the safe itself).
    • Emotional manipulation over a long period (romance scam elements).
  • Quote:
    • “The trigger that made her think it was a scam was the desire to send a safe. Physically send a safe.” (48:51, Joe)
    • “700K trunk box scam with extra steps.” (47:06, Joe)

---

Notable Quotes & Memorable Moments

• On AI Therapy:
  • “There's a mismatch between AI's goal to please and therapy's need for tough conversations and reality checks.” (23:50, Dave)
• On Paris Scams:
  • “Rule number one in any majorly touristed city is: anyone approaching you for any reason is not to be trusted.” (03:38, Maria)
• On Default POS Security:
  • “It should not be the burden of your local coffee shop to become an IT security expert.” (13:45, Maria)
• On Government Program Fraud:
  • “If you're going to spend $40,000 to help somebody find housing, you just go, 'I rented you this apartment.'” (35:10, Joe)
• On Recognizing Romance Scams:
  • “The trigger that made her think it was a scam was the desire to send a safe. Physically send a safe.” (48:51, Joe)
• Classic Wisdom:
  • “A computer can never be held accountable. Therefore, a computer must never make a management decision.” (29:43, Maria quoting IBM)

---

Timestamps for Important Segments

• Tourist Scam Story: 00:57–03:19
• POS Terminal Fraud Discussion: 05:53–14:18
• Main Segment – AI Therapy Study: 17:02–29:43
• Medicaid Housing Fraud: 31:28–43:59
• Catch of the Day – Romance Scam: 44:18–49:10

---

Tone & Approach

Conversational, witty, and approachable—while still respectful of the seriousness of the topics. The hosts regularly employ humor, personal anecdotes, and gentle self-deprecation to engage listeners. There is also a strong emphasis on empathy for the victims of scams and thoughtful discussion around the unintended consequences of technology and policy.

---

Summary Takeaways

This episode is a powerful reminder of the diverse forms deception, fraud, and social engineering can take:

• Whether via old-school tourist scams, vulnerable business technology, romance cons, or misuse of massive public benefit programs, bad actors will persistently probe for gaps in awareness and safeguards.
• Newer technologies, like AI therapy chatbots, pose unprecedented challenges, revealing how human reliance on machines can open doors to both hope and harm.
• Thoughtful scrutiny, skepticism, and maintaining human touchpoints remain key tools—and much needed antidotes—against a perpetually evolving threat landscape.

Listeners are reminded to:

• Stay vigilant, question anything that seems “off,” and check in on friends and family, especially those turning to new technologies for emotional support or handling sensitive financial transactions.

---

For more stories, advice, and resources about social engineering and modern scams, visit N2K CyberWire.

Transcript

[A]

You're listening to the Cyberwire Network, powered by N2K.

[B]

Hello, everyone and welcome to N2K, CyberWire's hacking humans podcast, where each week we look behind the social engineering scams, phishing schemes and criminal exploits that are making headlines and taking a heavy toll on organizations around the world. I'm Dave Buettner and joining me is Joe Kerrigan. Hey, Joe.

[C]

Hi.

[B]

Dav, colleague and host of the T minus Space Daily podcast, Maria Vermazes. Maria.

[A]

Hi, Dave. And hi Joe.

[B]

We've got some good stories to share this week. Before we do, we've got some follow up here. I guess I will start with this one. So this is from a listener who wrote in, I want to say their name is Khajetan. Do you think I have that right? Any guesses on pronunciation?

[A]

Kajatan.

[B]

Kajatan.

[C]

Kajatan.

[B]

Maria, got your time.

[A]

That would be my guess also.

[B]

All right. Apologies if I got it wrong, but they write in and say, hi, Dave, Joe and Maria. While listening to the latest episode and the statistics about travel scams. I remember one time some years ago when I performed a true miracle in Paris, France. While walking around Paris, I was approached by a woman with a note. She didn't say anything, just pointed at the note. The note stated in several languages that the woman was deaf and mute and they collected name signatures for support of the deaf and mute community in Paris. There were several names on the list already. Sure, why not, I thought, and I signed the list. After I did that, the woman unfolded the paper, revealing a new column with a donation value. Next to each name there was a donation of about 10 to €100. She handed the note back to me, pointing at the numbers and expecting money. Oh, I said gladly. I took the paper and crossed out my name. The mute woman went furious and started screaming, screaming at me in some language that was not French. And that was the time I performed a miracle and made a mute woman speak. Keep up the good work. I love your podcast, so well done.

[A]

Yes.

[B]

Yeah, my understanding is like, especially around the Eiffel Tower, like, it's just scammer central.

[C]

Right.

[A]

All of the tourist hot spots in Paris are pretty much scammer central. So a bunch of places I avoid

[C]

in Paris right now. I'm not worried about him though. He's. He's pretty sharp and pretty quick witted.

[B]

Right? Right. I mean, people are doing like the cups and balls tricks and the. I know, stealing lenses off of people's cameras. I've seen all those kinds of things.

[A]

So there's A, there's a common one, at least it used to be in Montmartre where the Sacre Coeur Cathedral, it has that beautiful view of the city where there are very aggressive touts that will try to tie a bracelet around your wrist as you walk by and then basically be like, I gave you a bracelet, you have to pay for it. And like, they can be very physically aggressive. So there are some parts of Paris that I just completely avoid. And that is one of them, frankly.

[C]

What happens if you get physical back? I wonder.

[A]

I'm a five foot one woman, so I don't wanna know.

[C]

Well, I have an idea. I have an idea. What happens is that like five or six of their friends show up and they all beat the crap out of you.

[A]

Yeah, that would, I would imagine. I don't wanna find out firsthand. So the answer is I just avoid those places. But I think rule number one in any majorly touristed city is anyone approaching you for any reason is not to be trusted.

[C]

Right. They're suspect.

[A]

So just keep walking.

[B]

It's so strange. Cause the Parisians are known for being friendly to Americans.

[C]

Right.

[A]

Know.

[C]

So yeah, that, that would raise my, my, my red flags immediately. Oh, I'm in Paris and someone's being nice to me. An American?

[B]

Yeah. I don't know, Joe. You think they'd peg you as an American? How long do you think that would take?

[C]

Oh, five seconds.

[A]

Yeah.

[C]

Everywhere I go and I've always been.

[B]

You'd be lumbering around with your baseball cap and. Or not your baseball cap, your cowboy hat. Cowboy hat.

[A]

Oh my God, they would love you, Joe, they would love you. I must be the one of the only Americans who had a really nice time in Paris and had no issues at all with Paris. I lived there for six months. I thought everybody was very nice.

[C]

Well, that's nice. I mean, you live there, they get to know you.

[B]

Yeah, I've only been there once.

[A]

I was the same person every time. All my inter, like my interactions were generally very nice. So I don't know. I did not have that rude Parisian experience at all. I thought everybody was great. But I also lived in New York City before then, so I was like, it's a city, people are city people here.

[B]

So I was going to say you're putting out all sorts of Greek vibes, so that was your camouflage.

[A]

Oh, I really wish that was the case. But I was very. I'm very obviously American. I'm very, very obviously American. I got clogged immediately, so. And it's fine. Yeah.

[B]

Every Attacker counts on one thing. Environments that Trust too much. ThreatLocker closes that gap with default deny at execution. Unknown software blocked. Trusted apps contained with ring fencing Configurations verified with Threat Locker DAC so you stay secure and compliant. ThreatLocker delivers the visibility and control CISOs need without adding operational pain, making zero trust real for teams of any size. Stop ransomware at its earliest point. Book a [e-mail] N2K. All right, well, let's move on to our stories here. Maria, you are up first this week. What do you have for us?

[A]

Well, I'm starting with some listener feedback because I thought this was a really interesting email and I wanted to follow up on their prompting. So here's the email we got. Hi, guys. Love the show. And that's the reason I wanted to use this email. I have a story idea about an issue that's happening here in the Great White North. We have lots of folks that pay by card when going to shops or restaurants. Many have these portal POS terminals and that's point of sale, everybody, not the other meaning of pos not like my card. Many of them, and many of them are left unattended. Shops have been defrauded because they haven't taken the time to chain the default access password on the terminal. Fraudsters are picking up the terminal and issuing unauthorized refunds to themselves, sometimes in the thousands of dollars. CTV is a national broadcaster here in the Great White north and he means Canada. I want to make sure I clarify for listeners and they have a story here and I, I, I'll follow up on that in a second. There are many other stories, but this is just one example, maybe worth investigating. Regards, Rob, AKA Crow Child Bob. Thank you, Crow Child Bob. So I, I went to the link that he sent and I did a little reading and, and I just, I thought this was a really interesting story because I had not been hearing much about this either. So those point of sale terminals in North America are frequently square clover toast. These have been common in a lot of the world for a long time. But, you know, you tap or swipe your credit card and they're pretty mobile. They're not necessarily attached to the counter, the counter anymore. You know, they do the transaction for you and they're the things that frequently they'll flip them around with the screen, say, Please give a 25% tip for this coffee that they just handed to you. That kind of thing. So these are those kinds of terminals Rob's referring to. And the story that he sent from CTV was piqued My interest because it starts off with a place called Souvlaki Hut in Toronto. So I was like, okay, this has got to be maybe in Greek town in Toronto. And a customer, a fraudulent customer, issued himself a $2,000 refund on a point of sale terminal at again, Souvlaki Hut, where maybe you're spending $4 a sale.

[C]

Right.

[A]

That is quite a refund.

[C]

$2,000 in souvlaki.

[A]

Aside from you, you have not met my family, Joe.

[C]

I love souvlaki. It's good stuff.

[B]

Can I just admit, I have no idea what souvlaki is.

[A]

Meat on a stick.

[B]

Meat on a stick.

[A]

Grilled meat on a stick.

[B]

Okay, all right, well that sounds like something I'd enjoy.

[A]

Grilled pork, grilled chicken, grilled beef, grilled lamb on a stick.

[B]

So it's like Greek shish kebab.

[A]

It is exactly, essentially, exactly like that.

[B]

Okay.

[A]

It's nothing. It is comfort food. Yeah, yeah, it's comfort food. So the, this specific situation with a two thousand dollar reef fraudulent refund, the, the criminal in this case picked up the terminal to hide what he was doing. So I guess he was doing this in semi plain sight of the store owner and then actually while still in the store gave himself that refund, which, that's kind of ballsy, I gotta say. And the owner's son, when he spoke to ctv said they had no idea that the terminal could do that without their explicit permission. So they didn't even know this was a thing, that this was possible. And a second business nearby to Souvlaki Hut in Toronto, the Pippins Tea Company had a similar scam happen to them. A guy pretended to buy a teapot from them and then refunded himself through the point of sale terminal $4900. That, that is, that is put the business out of business kind of money, depending on how small that business is. I mean, that is a lot.

[B]

Yeah.

[A]

So the interesting thing about the CTV story was a lot of the people they spoke to who are store owners, again, didn't know that this was possible, had no idea what protections were in place, if any, or what they should do to prevent it. Because in a lot of cases it's just there's a default setting or password in the point of sale owners. Those store owners don't know that they're supposed to change it. And this is a, for our friends in it, this is a familiar story for people who are familiar with like Iot in the home, we always say first thing you do when you bring it home is change that default password.

[C]

Right.

[A]

That information doesn't seem to have trickled to our friends in the retail world who are small business owners, especially in this situation. So security expert that CTV spoke to named Claudio Popa said that most of the POS terminals sold to small businesses are misconfigured from the start. So thieves are like, yay, this is great for me. So. And apparently, and apparently according to law enforcement in Toronto, this is a rapidly growing trend in the city, I'm sure in many cities all over North America, we're seeing this. And normally when we see point of sale fraud spoken about at least since 2020, especially like since COVID a lot of this discussions have been around ransomware or malware, these terminals being vulnerable that way. But it seems that the actual physical scam of going into the store and either giving yourself a refund while literally while the store's owner is turned, or. Or even breaking into the store after hours and accessing the point of sale terminal when nobody's there is. That's a. That's a big growing thing. So the recommendation to store owners is to make sure they lock away their terminals, change pins weekly, keep them out of reach when not in use. So really keep an eye on those terminals. And of course, change the default password. But do you know how to do that? Is it easy to do? Is it obvious how to do? Who knows? And the vendors and the victims in this case say that they really wish that the default passwords were safer, but again, they should change those. Or at least that they would require two factor authentication for things like refund limits. Which sounds like a good idea, right? So, yeah, it's to be clear, again, this is outright fraud, right? To give yourself a refund like this, and it makes. It just reminded me of a thing I think we did talk about a little while ago. The TikTok ATM hack that was going around where people were just hack. Do you remember this in airport?

[C]

Oh, I remember keenly.

[A]

Yeah. Yeah, this, this straight up check fraud that was going around virally on, on TikTok is like, hey, this is a way you can get free money from an atm. And people just were like, no, actually that's just fraud and that's a crime. Actually, Joe, I believe you said that that's just straight up a crime. I remember you saying, like, let's be clear, this is a crime.

[B]

That's one weird trick.

[A]

One weird trick to get yourself arrested immediately. So, yeah, it's just. It makes me think of that, like, this is. This is a crime. This is not like, one weird way to get money from a small business that really cannot spare that cash. So it's just very interesting that, you know, maybe as these terminals are getting hardened against cyber attacks, the physical attacks become the preferred vector, or all at the same time, who knows? But it's just very interesting. So thank you, Crow child Bob, for pointing this out.

[B]

You know, what I'm thinking about here that I never considered is that that little terminal, which in my mind I'm picturing something like a. That looks like an iPad, you know, and it's sitting there on the counter. And a lot of times the person behind the counter will enter in whatever they need to enter, and then they'll flip it around. Right. It's on some kind of side. And then walk away. Yeah. But even if it's attached to the counter, they'll flip it around and they may not necessarily be able to see what you're doing on the screen anymore. And so I think that if you know the. The guts of this thing and the secret incantations to get under the hood, there's your opportunity.

[A]

Yeah. Many times I've been in that situation and the person's busy. They're probably doing five things or the job of five people. So they flip that screen around and they walk far away and there's nobody else around. And there have been times in my life where I have been near a computer that is open or a terminal that is open. And I've often wondered when. What does this do? If I touch that button and I go, oh, I now have admin access to this thing. Isn't that fun?

[C]

I completely empathize with that. And sometimes I will actually have to put my hands in my pockets to make sure I don't touch anything.

[A]

Yeah, yeah. Some of us just naturally gravitate to that. And I know not to get myself into legal trouble. I obviously don't steal from anybody. That would be very wrong. But, you know, but sometimes I'm just like, it is so easy. And I just. It makes me a little scared and sad that these are not, and it should not be the burden of your local coffee shop to become an IT security expert. Like, these things should be locked down. So, yeah, it's just, oh, gosh, don't walk away, guys. Just keep an eye on them.

[B]

The other one I think of that I see all the time is if I'm in a large store like a Target or a Macy's or something like that, where there are People who are out there stocking shelves or rearranging things, checking, pricing, labeling, stock, all that kind of stuff, they will very often have a little handheld device that assists them in doing that. Maybe it prints labels, maybe it scans barcodes. They can put in the amount of inventory that there is out on the floor, that sort of thing, using this device. And you see this device, you recognize it for what it is, because there are probably only a handful of variations on this that retailers use. But, boy, do those stock people leave them lying around the store a lot.

[C]

Right?

[B]

You know?

[A]

Yeah, they do. Yeah.

[B]

Yeah.

[A]

And I know we shouldn't mess with these things. I understand that, but.

[B]

I know.

[A]

But I really want to play with one.

[B]

I know. I want to know. I know.

[A]

It's been sitting there, you know, shame on you.

[B]

Right? Press the button, you just leave it at the. What choice do I have? It's just laying there. There's no one around.

[C]

I was in one of our local stores one day and they had this, like, a tablet that normally would display something, but whatever app it was had crashed and left Windows, I think. Ce. Was that the edition that was like the Embedded Systems Edition?

[B]

Wince.

[C]

Yeah, Yeah. W. I Wince?

[B]

Yeah.

[C]

Is that what it was called?

[B]

Well, that's what some of us called it.

[A]

Right.

[C]

And it's just sitting there, and I'm like, what's going on here? I pushed the start button. Now, the screen didn't work. The touchscreen did not work. So I was thwarted. But if I had my wireless or my keyboard with me, I could have just plugged it in. One of the USB ports, it was just sitting there exposed, Right, right.

[B]

I don't know, Joe. I expect if you're walking around with your usual backpack, you have a keyboard in there, right?

[C]

Yeah, I might. Sometimes I do.

[A]

Maybe an antenna. Maybe some other things.

[C]

There's an antenna sticking out of my bag, right?

[A]

Oh, well, yeah, yeah, yeah. When you go to defcon. No, I'm just kidding.

[C]

I gotta go to defcon.

[A]

How popular are you when you go to DEF con?

[C]

I've never been.

[A]

Really?

[C]

Yeah. I want to go.

[A]

No.

[B]

Bad Joe. No. When Joe's backpack is so big that when he's driving around in his car on the highway, he has to pull over into the way stations.

[C]

Right.

[B]

Because of how much stuff he has in his backpack.

[A]

How's your scoliosis, Joe? Is it?

[C]

Well, the backpack pulls it straight.

[A]

Oh, there you go. Okay.

[B]

Yeah. All right. We will have a link to that story in the show notes. I'm up next here. And My story comes from the folks over at Ars Technica, and this is about people using large language models, the AI systems, for therapy. Before we dig in here, let me ask either of you, do either of you ever interact with these models in a purely conversational kind of way?

[C]

I try to, yes. Although I've found lately that ChatGPT has changed the voices to make them hesitate and use inarticulate speech like, and effort to make them seem more relatable. And it does not work with me. I don't like it.

[B]

Oh, interesting.

[C]

Yeah.

[B]

Okay.

[A]

Maria, I just want to ask a clarifying question about your question. Are we talking about mental health models specifically or just.

[B]

No, no, I'm just saying in, like, you ever just, you know, strike up a conversation to see how it does or. You know, I've heard of people who just, like, they'll. They'll kill time during their morning commute just by chatting with the AI, you

[A]

know, I do not. But I think that's because in the 90s, when AI, not AI, these are just regular old AOL chatbots and the like were around, I think I got that out of my system back then, and I just. I didn't find it very fulfilling. So that temptation is not there for me anymore. I know they're much more sophisticated, but I just. I don't know, I just don't feel curious about that. I don't know why.

[B]

Yeah, I haven't either, any more than just playing around with it deliberately, you know, here at my desk. I haven't used it for conversational time killing, anything like that.

[C]

Well, I want to clarify what I said. I don't use it for conversational time killing, like, and asking, like, just, hey, how you doing? I have a specific question in mind and I'll ask clarifying questions and drill down on it in a conversational manner.

[B]

I see.

[C]

But I don't know that I've ever actually sat down and talked to it like a person.

[B]

Right. Well, lots of people do for better, for worse. And some researchers at Stanford recently presented at a conference, this was the ACM conference on Fairness, Accountability and Transparency. And they presented on how these large language models do when responding to people who have mental health conditions. And they tested the AI systems using fictional scenarios that reflected serious symptoms, things like suicidal ideation and delusional thinking. So some heavy, serious stuff. And what they found is that the models often failed to follow accepted therapeutic guidelines, and they would sometimes validate harmful beliefs or offer advice that could make things worse. And this isn't just the General chatgpts of the world. This is also the chatbots that are designed to provide therapy. Right. They're supposed to be tuned.

[A]

Oh, my God.

[C]

I don't think that's a good idea to have a chatbot that's designed to provide therapy.

[A]

But a lot of people use them, Joe. A lot of people who can't afford or cannot find a mental health provider in person, they. They rely on these. I know several people who do. And, oh, boy, I've got some phone calls to make after this episode because I did not know this. That's alarming.

[B]

Yeah. I mean, and to be fair, I mean, I think there are plenty of people who have. Who find serious value in this. Right. Legitimate value. That it is someone to talk to. It's someone who's not going to judge you. It's always available. I remember early on, Joe, you and I have talked about this way back on this show. There was a story when Siri first came out for the iPhone that there was a young man who had some developmental issues.

[C]

I think it was autism. He's on the spectrum.

[B]

And the thing that was so great about Ciri is that she had endless patience. Right. This kid was basically a motor mouth and was just talk, talk, talk, talk, talk, and was wearing the rest of his family out. You know, God bless them. I mean, they were doing their best and they were trying, but the bot had endless patience. And so in that case, it was a good thing for everybody. And so obviously, as these things have gotten more advanced and more capable and more real seeming, there are several that have been spun up that are designed to help you with in a therapeutic kind of way. But this study found that they don't always do well. They said they tend to validate harmful beliefs which they call sycophancy, like, I guess, sycophants.

[C]

Sycophancy.

[A]

Yescophancy. Sycophancy.

[B]

Sycophancy, yeah. Sycophancy potentially reinforcing people's delusions or conspiracy theories.

[C]

You are right, Joe. Wow.

[B]

Yeah. Everyone is out to get you.

[C]

Right?

[A]

That tinfoil hat of yours looks great, right?

[B]

Yeah. And there are cases in the media where this has contributed to suicide or fatal incidents. Where there was a case where a man who had schizophrenia was killed by police after interacting with ChatGPT.

[A]

Oh, my gosh.

[B]

They're putting the sort of the warning flag up about this. A couple other little details. They said that it doesn't seem like the newer or larger models are any better at this than the older, smaller models were. They're looking for critical evaluations and better safeguards. But they're saying that they don't want to abandon the notion of AI in mental health, that there is a part that it could play, but we just need to be able to put better guardrails on it. And this was the real take home for me. They said there's a mismatch between AI's goal to please and therapy's need for tough conversations and reality checks. And to me, this really tracks with. Because my experience with these AIs is like, you know, hey, ChatGPT, the sky is red. And the ChatGPT will say, oh, Dave, you're right. You're so smart. The sky is red.

[C]

Right? Yeah. I've had that happen with me as well. Right now. Normally I am right about something in ChatGPT is, what do they call them? Confabulations. Now, they don't want to call them hallucinations.

[B]

Yeah, confabulations.

[C]

And ChatGPT will be confabulating about something, but I'll correct it. There was one time I asked it who invented something, and knowing full well it was me, and I hold the patent on it. But

[B]

this is a ChatGPT version of Googling yourself.

[C]

Yes, exactly.

[A]

Who is the brilliant person who made the thing?

[B]

It came back with the singular mind from which this invention was spawned.

[C]

Right. But here's the thing.

[A]

Yes. When it's phoned.

[C]

Here's the weird thing about that conversation is after. After I gave it the patent number to go look at, it went, oh, that's you. And I had. You know, I don't know whether I've ever conversationally told ChatGPT who I am.

[B]

Oh, but it knew.

[C]

But it knew.

[A]

You can check the. Its memories, though. You can check to see what it knows about you so you can find them out.

[C]

Well, the. The company OpenAI has my. Has my information. They have my billing address and all that stuff because I do pay 20 bucks a month for that service. I think it's worth it. I get a lot of use out of it. And it really does help. At my office the other day, somebody actually accused me of being someone who hates not knowing something, which is 100% correct. I cannot stand not knowing something.

[B]

I was gonna say that tracks

[C]

when I am in the car with my son or my wife and I'm driving, which I usually am, and I have an idea. We're having a conversation and I want to know something. They both take great delight in going, yeah, I don't know. And then doing nothing else.

[B]

Oh, they Hang you out to dry.

[C]

They hang me out to dry. Right.

[B]

Intentionally. And I'm like, oh my, what torture.

[C]

Can you, can you Google that and read to me what it says?

[A]

They know that button depressed you. They know.

[B]

Exactly. It's diabolical.

[C]

Joe, I am not your. But now I don't even. I just turn on ChatGPT in voice mode and I ask it a question and I get my answer, right? And I go there so that when

[B]

the day comes, you get an answer. When the day comes that you're up in the, in the clock tower with a high powered rifle and people are asking what happened? What possibly could have made mild mannered Joe turn to these evil actions that

[C]

might be the first time anybody's ever called me.

[B]

You'll be up there screaming. They wouldn't give me the answer. They wouldn't give me the answ.

[C]

I mean, that is one of the things I'll tell you, Dave, that will set me off in a, in a, in a conversation. That will set me off in, you know, if somebody just says, you know, unless there's some kind of reason for not knowing, like, hey, that I can't tell you that because it's proprietary information or I can't tell you that because it's, you know. You know, I have a lot of friends who work in defense. I can't tell you that. It's classified. I can't tell you that for xyz. A good reason. Okay. I can't know it even. Hey, I can't tell you that because it's personal. Yeah, okay, that's fine. I can respect that. But I just don't want to tell you or I'd rather you didn't know that information and just remained in the dark about it. And generally I'm talking about general information. Right. You know, I wanted.

[A]

Not what happened before the big bang kind of existential questions, right? Like this is torturing you day and night to know.

[C]

Yeah, I mean, well, actually, what happened before the big bang? That would be a good question to have an answer to. Unfortunately, we don't have an answer to that and probably never will.

[A]

When you get an answer to that, Joe, please share it with me. Nobody else I would very much like to know.

[B]

When I was in high school, I had a buddy who I would call regularly to call up your friend to get together, check on things. This is how we used to do it back then, before we had mobile devices. And my buddy had a younger brother who was quite a character and pretty subversive little lad. And so sometimes I remember one time I called and I said my friend's name was also David, and his little brother answered the phone and I said, hi, Matthew, this is Dave. Is your brother David there? And he said, no, he's not here. And I said, well, do you have any idea where he is? And he said, I can't tell you that. And I went, what? He said, no, I can't tell you that.

[C]

Okay, what did he mean? Did he mean that? He didn't immediately. Now I'm right, you've already got me. I'm over here wiggling in my chair. Dave.

[A]

Right, right.

[C]

What do you mean you can't tell me that?

[B]

Yes, exactly. I said, well, why can't you tell me? He said, I can't tell you because. I don't know. I was like, you jerk.

[C]

Right, well, why is that? Yeah, exactly.

[B]

Exactly. Next time I'm over there, I'm going to give you a Dutch rub or something. You got a gotcha coming.

[C]

You got an atomic wedgie coming.

[B]

Yes, exactly, you little smarty pants. Yeah, anyway, so I can relate, Joe. I can relate. That's not a fun position to be in, especially when somebody's yanking your chain.

[C]

Yes.

[B]

All right, so back to the story here, which of course we will have a link to in the show notes. Just be mindful, like Maria said, check in with your friends if you know they're using these things and just let them know. Remind them that if things get dark and it seems as though one of these models is leading them down a bad path, that they do have real life human friends that they should check in with and have their back. Because evidently the guardrails on these things are not quite robust enough to be fully trusted.

[A]

Yep. Reminds me of that IBM slide. Do you know the one that's very popular in AI discussion circles right now? The one from 1979? Yeah. The acute computer can never be held accountable. Therefore, a computer must never make a management decision. So in this case, like a mental health decision, Right? Yeah, right. I always think back to that one.

[B]

No, it's good. All right, I'll tell you what, let's take a quick break here to hear from our sponsor. We will be right back. Most environments trust far more than they should and attackers know it. Threat Locker solves that by enforcing default deny at the point of execution. With Threat Locker allowlisting, you stop unknown executables cold. With ring fencing, you control how trusted applications behave. And with ThreatLocker DAC defense against configurations, you get real assurance that your environment is Free of misconfigurations and clear visibility into whether you meet compliance standards, ThreatLocker is the simplest way to enforce zero trust principles without the operational pain. It's powerful protection that gives CISOs real visibility, real control, and real peace of mind. ThreatLocker makes zero trust attainable even for small security teams. See why thousands of organizations choose ThreatLocker to minimize alert fatigue, stop ransomware at the source, and regain control over their environments. Schedule your [e-mail] N2K today. And we are back. Joe, it is your turn. What do you got for us this week?

[C]

Before I get to my story, I want to talk about something. My mom texted me yesterday and she said, hey, I got this message that says, I owe toll money. I was a scam. Don't even. Don't do it.

[B]

Yeah.

[C]

Then today I got one and it was sent to, like, a group of, like 10 people.

[B]

Oh, yes.

[C]

I don't know if you've ever seen that happen in these toll scams.

[B]

I recently got a group text toll scam. Yeah.

[C]

I responded to the text and said, you all probably already know this, but this is a scam. Do not click on any of the links.

[B]

Yes.

[C]

So hopefully that went out to everybody.

[B]

Yes, yes, I have gotten those.

[C]

Yeah, I don't know if that's helpful. I don't know if that may have just wrote myself in to more of these things. Who knows? Oh, this is a good number. And this guy, like, helps me.

[A]

I don't know if I respond, but thank you for your sacrifice, I guess.

[C]

Yes, I guess.

[B]

On behalf of the rest of us,

[C]

on the behalf of the other 10 people in that group. So my story actually comes out of K A R E. You know, west of the Mississippi. Radio stations and television stations begin with the letter K. Yes. I find that very difficult to deal with. Having grown up entirely and spent most of my life on this side of Mississippi. I want to say W A R E, but it's not. It's K A R E, Channel 11 in Minneapolis. Coming from AJ Lago, I hope I'm saying that right. Lego L A G O E. Kelly Dietz and Gary Knox.

[B]

Okay.

[C]

And in 2020. 2020, Minnesota became the first state to offer a Medicaid benefit called Housing Stability Services.

[B]

Okay.

[C]

Okay. Now, this is a Medicaid benefit aimed at helping people avoid homelessness, typically targeting, like, older people, people who are dealing with addiction, maybe going through addiction treatment, and maybe some disabled people. When Minnesota planned this, they estimated that it was going to cost about $2.6 million a year, which doesn't sound like a lot of money. $2.6 million a year. I can make sure that people who need medical attention don't go homeless. Okay, sounds like a good idea. Good investment.

[B]

Sure.

[C]

First year, first full year of billing. In 2021, the billing was $21 million, almost 10 times what they anticipated. In 2024, the billing was $144 million. Now you are talking something that is two orders of magnitude bigger than what was originally planned. I mean, and I know every time there's a government program, it doesn't matter which organization in the government is. Is. Is doing it. Department of Defense, Department of Health and Human Services. They all have this number that they tell you, and then a couple years down the road, that number is much bigger. But this is unprecedented in terms of, you know, you're not talking about a small increase. You're talking about two orders of magnitude in four years. So how is this possible? Well, there is an acting U.S. attorney, his name is Joseph Thompson, and he has an answer. He says Minnesota has a fraud problem and not a small one.

[A]

Gee, yeah.

[C]

Federal agents laid out examples of these. Of huge bills being done by some of these companies who are doing hss. That's what I'm going to call it. The. What is it called? Housing. Housing Stability Services. I'm just going to say HSS from now on, where they're reportedly receiving, like, large sums of money, like 40 and $52,000 for clients in a year. 40 to $52,000. Helping clients find housing.

[A]

Hmm.

[C]

So, I mean, really, if you're gonna spend $40,000 to help somebody find housing, you just go, I rented you this apartment. Here it is.

[B]

Right, right.

[C]

Live here. And, you know, it'll cost less than $40,000 to do that. But they. They executed a warrant, a search warrant on these companies. And according to the warrants, these companies promise to help these individuals find stable housing as they finish their treatment programs. All too often, the companies do not actually provide any real assistance to. In finding houses. Instead, the companies simply use the information provided by their recruited clients to bill Medicaid for housing stabilization services they do not actually provide. So the Federal investigation identified 22 different HSS providers in this particular warrant. And they said that they served a warrant on these companies, many of them in the same building.

[B]

Hmm.

[C]

Right. So they all have the same address, just different suite numbers in the address. And these providers collectively received more than $8 million in Medicaid payments for housing stabilization services from January to May. January of last Year to May of this year. So in 18 months, almost 18 months, they have exceeded the original planned cost in fraud by like three times. Amazing. And then they have these companies, they list them out. I'm going to name them. This is all, of course, alleged, but because it's a warrant and a search that's going on. Brilliant Mind Services, Leo Human Services, Liberty Plus Pristine Health and Falad Care. F A L A D Care Incorporated. In one case, brilliant minds billed $2,000 for services they claim to provide to a woman named Rachel. But Kare actually previously interviewed Rachel because this has been on their radar for a little while, apparently. And she said she never received any services from the company except for one occasion when a person named Mohamed Mohammed dropped off a shark vacuum cleaner at an apartment that she had found for herself without the help of this company. So, hey, thanks for everything. Here's a vacuum cleaner. I did something

[A]

terrible.

[C]

Yeah. But even after this happened, CARE K A R E. Not gonna call them CARE because they're actually a broadcast station. That's call letters, right?

[B]

Yeah.

[C]

Right, Dave. That's a call sign.

[B]

How much you want to bet that their local promos are 11 cares?

[C]

It might be.

[A]

Yeah, of course, of course.

[B]

Right.

[C]

We can't do that with our h. With our amateur radio call signs, though.

[B]

No.

[C]

Yeah. Kare discovered that Brilliant Minds continued to build Medicare for additional house hunting services that they probably never provide. And there's one author that says in one of Rachel's entries in the system, I visited these properties. I went. Went in person on Rachel's behalf because these two options were strong matches with her housing criteria. It's a scam. It's a total scam, says Rachel. Rachel's Medicaid billing showed that another company, Leo Human Services, which is one of the companies in the list above, also billed the HSS program. And Rachel says, I never even interacted with those people. So they're sharing information. They're billing the government back for these services that they're not providing. And then they're telling their buddies, which probably all just one organization, right, that they've stood up all these different. These different companies, and then now they're passing the information, hey, I just billed Medicaid for $2,000 for Rachel over here. Pass it off to Leo. Leo will do it for another $2,000, and we could pass it off to the next guy, and everybody makes bank. So, yeah, this is a huge problem. In 2025, Minnesota passed a new law that has more stringent requirements on it, but it was. It's amazing to me that within one year of passing a law, there's already so much fraud going on in this, in this system that, that it's four times what, you know, almost 10 times actually. What, what you'd expect to pay.

[A]

Well, it, it feels like anytime there's a new program, the, the scams come up immediately. Like. But the COVID ppp, the, those funds. How many of those. Yeah. How much fraud happened there?

[C]

Tons.

[A]

And they're still, I think they were still chasing it down. They're still chasing it down now, five years later.

[C]

Oh, yeah.

[A]

It's the whole time you were describing this. That's what I was thinking of.

[C]

Yeah, that's a good point. I was saying that hopefully those people all get prison time. You know, these are, these are taxes.

[A]

I think a lot of people got away with it.

[C]

I think you're right. I think you're right and that's unfortunate. I have heard of people getting, facing serious ramifications for the PPP loans that they weren't entitled to. On the other side of this, there's the issue of like with the PPP loans, there was a time sensitive issue there. Now with this law, I don't know that there is a time sensitive issue for $40,000 in services being billed. I don't buy that. Maybe there's a time sensitive issue. Okay, so this person needs to, to pay rent this month. What's their rent? The rent is not $40,000. It isn't. There's no way.

[B]

Right.

[A]

When you have housing involved, though, I imagine I'm just, I'm thinking, I mean, obviously fraud is bad. Let's put that out there. I'm thinking if I was trying to administrate a program like this where it's like, hey, this is giving money for people who need help getting housing when housing is always so hard to come by, do we want to make this move as slowly as possible, which means it could be years before somebody gets housing help, or do we want to sort of hand it out first and then retract it later if fraud has happened? Rather err on the side of helping somebody than a little more, than a little less. I'm sure there's a lot of debate there, honestly, but I imagine that might be part of the struggle is like you have to act in a timely way. Right?

[C]

That is absolutely part of the problem.

[B]

While you were talking, I was trying to find some statistics here to try to figure out because every program is going to have a certain amount of fraud, just like every store is going to have a certain amount of shoplifting or damaged goods or those kinds of things. And the charts that I'm seeing here say that most public services have a built in tolerance for between 1 and 3% fraud as acceptable risk. But when rates get above about 5%, that is when internal audits are usually triggered or policy reform or things like that. But seems like under 10% is typical for these sorts of things.

[C]

Right.

[B]

And I think Maria makes an excellent point. You know, how much fraud do you accept in exchange for not having too much friction for the people to get the services they need? And that's not always an easy answer.

[C]

That is an excellent question when you're talking about homelessness.

[A]

Yeah.

[C]

I mean, cause that will be a devastating impact to somebody. But yeah.

[A]

And you're talking, especially in the north when it gets cold, people could die, you know, with exposure. I mean, it's like this can be life or death.

[B]

Yeah, yeah. So it's good that they're going after the fraudsters and not the people who are being, you know, falsely, whose names are being put in to falsely claim these funds.

[C]

Right, Yeah. I think that's an important distinction. You know, these people are in their own right, victims, first off, they've got some, some kind of health malady. And I'm going to go on, I've said this a lot of times in this country. We have to stop treating addiction like it's a morality issue. It's got to be treated like a healthcare issue. It really does. And you know it. When you, when somebody is, is addicted to a substance that is physically addictive, they just, they can't just stop. It may kill them. Doing that may kill them. So there has to be treatment options for these people. So I don't have a problem with treating people who are dealing with some kind of substance abuse issue. But at the same point in time, I think that the people who are caught for these, I think the punishment for people who have defrauded this program should be severe. Very severe.

[B]

Yeah, I'm for that. All right, well, again we'll have a link to that story in the show notes. Joe, Maria, it is time to move on to our catch of the day.

[C]

Dave, our catch of the day comes from the scam subreddit.

[B]

So this is someone who had an ongoing, let's call it a relationship with a scammer. And they wrote up a description of this. They said a patient scammer finally asked me for money after five months of text and calls claims that they want to send me, $700,000 profit from selling property because his funds are frozen as part of an inheritance lawsuit. So we've certainly heard of those before, right?

[C]

Absolutely.

[B]

Sent me a suspicious video of a safe. Look at all this money in here. Yeah. Just inside this safe.

[A]

Interior or exterior?

[C]

Right.

[B]

I'm gonna go with exterior. Right. I'm just gonna, you know, send me a photo of Fort Knox and said, this could all be yours. And then a tracking number from a shipper with misspellings. Of course, the package never came. Then he called and sent me a screenshot from the shipper saying they needed €15,000 for customs. I already knew this was a scam. As soon as they wanted to send me a safe. Doesn't seem like the most efficient object to send because safes are heavy.

[A]

Yeah, I was gonna say international shipping weight kind of a thing. Yeah. Safe.

[B]

Yeah. He says, I'm disappointed I became emotionally invested in a fake, but at least I never gave them money or account numbers.

[C]

Yep.

[B]

So this message reads, and it says, Dear Ms. Nicole, we hope this message finds you well. We're writing to inform you that your shipment from Berlin, Germany to Cleveland, Ohio, usa, has arrived and is currently undergoing customs clearance at the US port of entry. As part of the clearance process, US Customs has assessed a total of €15,000 in duties and associated handling fees.

[C]

Hold on right there. Why is U.S. customs accepting payment in euros?

[B]

Excellent question.

[A]

Don't ask. Don't ask questions. Don't ask questions.

[B]

Joe, do you want the $700,000 or not?

[C]

I want the $700,000.

[B]

Well, then stop asking so many questions. To avoid any delays or additional storage charges, we kindly request that this amount be remitted at your earliest convenience. If you have any questions regarding the payment or require assistance with the process, please don't hesitate to contact our team directly. We appreciate your prompt attention to this matter and thank you for choosing Netlight Shipping. And it's warm regards, Netlight shipping GmbH. Which is German for incorporated, right? I think so, yeah, something like that.

[A]

Yes.

[B]

Yeah, it's a stamp for German companies, customer service department. So, you know, there's obviously trunk box

[C]

scam with extra steps.

[B]

Yes, exactly. I like that. In this request for the €15,000, they snuck in this thing about avoiding any delays or additional storage charges. So act now. It's going to cost you even more money.

[C]

The artificial time horizon.

[B]

That's right.

[C]

And they let you fill that in, right? They don't define it for you. They let you worry about that.

[B]

Yeah, yeah. So 700.

[A]

The attention to detail, though, is interesting. Like the. The phone number is definitely a German area country code. I wonder if we put. Not that we should or would, but I wonder if the other Maria is Megan McAll.

[B]

As we all know, Maria plays pretty fast and loose with the links. I mean, I don't want to say anything, but word around town, Maria, is that you're pretty fast and loose with the Lynx.

[A]

I just love putting my hand on that hot stove over and over. Yeah, right. The attention to detail throughout this, like, everything looks convincingly like it should be from or to Germany. So a lot of times it's more messy than that. But this actually looks consistent there. Except for the, oh, Cleveland, Ohio, USA. €15,000 thing that Joe picked up on him.

[C]

Right?

[B]

Yeah, but imagine five months of text messages and calls between this person and the scammer and it finally got to this point for them to cut it off.

[A]

Yeah, I'm glad they couldn't. They weren't leading them on. They were actually taken for those five months. Sounds like so. That's right.

[B]

It says, I'm disappointed. I became emotionally invested in a fake. So this was partially a romance scam.

[C]

Right. But the. Probably the interesting part is that the trigger that made her think it was a scam was the desire to send a safe. Physically send a safe.

[B]

Yeah.

[C]

That's how she knew it was a scam, which is good. Good that she picked up on it. You know, she got. She. She got off easy.

[B]

Yeah, yeah, absolutely. All right, well, we would love to hear from you. If there's something you'd like us to consider for the show, you can email us. It's hackinghumans2k.com. Most environments trust too much and attackers know it. Threatlocker enforces default deny at execution. Blocks unknown apps and limits what trusted apps can do. Stop ransomware at the source. Get your [e-mail] N2K. And that is hacking humans. Brought to you by N2K CyberWire. We'd love to hear from you. We're conducting our annual audience survey to learn more about our our listeners. We're collecting your insights through the end of August. There's a link in the show. Notes, please do take a moment and check them out. This episode is produced by Liz Stokes. Our executive producer is Jennifer Ibin. We're mixed by Elliot Peltzman and Trey Hester. Peter Kilpe is our publisher. I'm Dave Bittner.

[C]

I'm Joe Kerrigan.

[A]

And I'm Maria Vermazes.

[B]

Thanks for listening, Sam.