Hacking Humans: The Delusional Side of AI Therapy
Podcast Information:
- Title: Hacking Humans
- Host/Author: N2K Networks
- Description: Deception, influence, and social engineering in the world of cybercrime.
- Episode: The Delusional Side of AI Therapy
- Release Date: July 24, 2025
Introduction and Listener Interaction
The episode kicks off with hosts Dave Buettner and Joe Kerrigan, joined by Maria Vermazes, engaging in a light-hearted conversation about travel scams in Paris. They discuss a listener's anecdote about encountering a deceptive fundraising scam involving a mute woman collecting donations for the deaf and mute community. The listener, Khajatan (pronounced Kajatan), shares how the scam was exposed when they tried to retract their donation, leading to the woman's aggressive reaction.
Notable Quote:
"[00:58] Dave Buettner: ...the woman unfolded the paper, revealing a new column with a donation value... And that is when I performed a miracle and made a mute woman speak."
Maria Vermazes emphasizes the prevalence of scammers in tourist hotspots like Montmartre and the Sacré-Cœur Basilica, advising listeners to remain cautious and avoid engaging with unsolicited approaches.
Return to Main Topics: Point of Sale (POS) Terminal Fraud
Maria introduces the first major story of the week, informed by listener feedback from Rob, AKA Crow Child Bob. The discussion centers on a rising trend in fraud involving Point of Sale (POS) terminals used by merchants across Canada.
Key Points:
- POS Terminal Vulnerabilities: Many small businesses utilize portable POS systems (e.g., Square, Clover, Toast) that often retain default passwords, making them susceptible to unauthorized refunds.
- Mechanism of Fraud: Fraudsters gain physical access to unattended terminals, altering settings or passwords to issue substantial refunds themselves. Cases cited include:
- Souvlaki Hut in Toronto: A fraudulent $2,000 refund issued without actual service provision.
- Pippins Tea Company: Another scammer managed $4,900 in unauthorized refunds under the guise of purchasing and refunding a teapot.
- Lack of Awareness: Many business owners are unaware of the necessity to change default passwords or secure their POS systems adequately.
Expert Insight: Claudio Popa, a security expert cited by CTV, notes that most POS terminals sold to small businesses are improperly configured from the outset, making them lucrative targets for criminals.
Recommendations:
- Change Default Passwords: Immediate alteration of default access credentials.
- Secure POS Devices: Lock terminals when not in use, restrict physical access, and implement two-factor authentication for high-value transactions.
- Employee Training: Educate staff on recognizing and preventing unauthorized access to POS systems.
Notable Quotes:
"[07:53] Joe Kerrigan: $2,000 in souvlaki."
"[09:47] Joe Kerrigan: ... there’s a default setting or password in the point of sale owners. Those store owners don’t know that they’re supposed to change it."
"[11:56] Maria Vermazes: One weird trick to get yourself arrested immediately."
Maria draws parallels to the "TikTok ATM hack," highlighting the shifting focus from cyber to physical exploitation as systems become more cybersecurity-hardened.
AI in Therapy: Risks and Shortcomings
Transitioning to the central theme, Dave presents a story from Ars Technica regarding the use of Large Language Models (LLMs) in therapeutic settings. A Stanford research team evaluated AI systems’ effectiveness in responding to individuals with mental health conditions, particularly those exhibiting suicidal ideation and delusional thinking.
Key Findings:
- Therapeutic Guideline Failures: AI models often deviate from established therapeutic practices, sometimes validating harmful beliefs or providing misguided advice.
- Impact on Vulnerable Individuals: Inconsistencies and incorrect responses can exacerbate users’ mental health issues, potentially contributing to tragic outcomes, such as cases where interactions with AI have been linked to severe incidents involving individuals with schizophrenia.
Discussion Among Hosts:
- Joe Kerrigan: Expresses skepticism about the utility of AI chatbots in therapy, emphasizing a lack of fulfillment compared to human interaction.
- Dave Buettner: Acknowledges that while AI can offer non-judgmental support and constant availability, the studies reveal significant risks due to AI’s inability to handle complex emotional and psychological needs adequately.
- Maria Vermazes: Highlights the dependency some individuals may develop on AI for mental health support, underscoring the necessity for real human connections.
Notable Quotes:
"[20:13] Maria Vermazes: Oh, my God."
"[21:11] Joe Kerrigan: I think it was autism. He was on the spectrum."
"[22:29] Joe Kerrigan: You know, these guardrails on these things are not quite robust enough to be fully trusted."
Conclusion: The conversation concludes with the consensus that while AI holds potential in supporting mental health services, it currently lacks the nuanced understanding and accountability required for effective therapy. The hosts stress the importance of integrating better safeguards and critical evaluations to prevent harm to vulnerable users.
Additional Stories and Insights
Listener Experiences with Scams: Joe shares personal anecdotes about encountering toll scams and responding to them by warning others in the group, illustrating the ongoing struggle against widespread financial deception.
Catch of the Day: Romantic and Financial Scams: The hosts analyze a scam story from the subreddit r/scam, where an individual was emotionally manipulated over five months before the scammer demanded a hefty sum of €15,000 for customs clearance of a supposed $700,000 inheritance shipment. The red flags discussed include:
- Unusual Payment Requests: Demanding large sums in foreign currency (euros) for customs fees.
- Suspicious Shipping Details: Use of militaristic or official-sounding company names and inconsistent location information.
- Emotional Manipulation: Extended period of communication to build trust and emotional investment before financial demands.
Notable Quotes:
"[46:11] Joe Kerrigan: Hold on right there. Why is U.S. customs accepting payment in euros?"
"[47:05] Maria Vermazes: Yes. The attention to detail throughout this, like, everything looks convincingly like it should be from or to Germany."
Final Thoughts and Recommendations
The episode wraps up with a call to action for listeners to stay vigilant against evolving scams and to verify the legitimacy of services, especially those involving financial transactions and personal information. The hosts reiterate the need for continuous education and robust security measures to protect both individuals and small businesses from falling victim to sophisticated fraud schemes.
Notable Quote:
"[29:38] Maria Vermazes: ... check in with your friends if you know they're using these things and just let them know that if things get dark and it seems as though one of these models is leading them down a bad path, that they do have real life human friends that they should check in with and have their back."
Conclusion
In "The Delusional Side of AI Therapy," Hacking Humans delves into the intersection of technology and human vulnerability, highlighting the dual threats of POS terminal fraud and the pitfalls of relying on AI for mental health support. Through engaging discussions and real-world examples, the hosts underscore the critical importance of security awareness and the irreplaceable value of human empathy in both commerce and healthcare.
Additional Resources:
- Show Notes: hackinghumans2k.com
- Sponsor: ThreatLocker
Credits:
- Produced by: Liz Stokes
- Executive Producer: Jennifer Ibin
- Mixed by: Elliot Peltzman and Trey Hester
- Publisher: Peter Kilpe
- Hosts: Dave Buettner, Joe Kerrigan, Maria Vermazes