Phishing Tests Are Getting Downright Mean

Summary

WSJ Tech News Briefing: "Phishing Tests Are Getting Downright Mean" (February 10, 2025)

Host: Pierre Bienname
Producer: Julie Chang
Supervising Producer: Kathryn Millsop
Guest Contributors:

• Bob McMillan: Wall Street Journal Computer Security Reporter
• Bell Lin: Enterprise Technology Analyst at the Wall Street Journal

---

1. Introduction

In the February 10, 2025 episode of WSJ Tech News Briefing, host Pierre Bienname delves into two pressing issues in the tech world: the escalating sophistication of phishing attacks and the challenges posed by artificial intelligence (AI) hallucinations. The episode features insightful discussions with experts Bob McMillan and Bell Lin, shedding light on both cybersecurity threats and advancements in AI reliability.

---

2. The Escalating Threat of Phishing Attacks

Phishing Defined and Its Evolution

Phishing remains a prevalent cyber threat, serving as the initial vector in approximately 14% of data breaches last year, according to a Verizon analysis cited by Bienname at [00:19]. Bob McMillan elaborates on the mechanics of phishing:

"They try to play in your mind. They try to get you in some kind of panic mode. So usually what happens with these phishing emails is there's some very, very important piece of information they promise..." ([01:29]).

Sophisticated Phishing Tactics

Modern phishing schemes have evolved beyond clichéd scams like the "Nigerian prince" emails. McMillan highlights how attackers now mimic legitimate corporate communications to deceive targets effectively:

"The hackers are getting very clever. They know how corporations work and they know which kind of emails are high priority..." ([04:03]).

These advanced tactics often involve spoofing emails from high-ranking officials, such as CEOs, especially during critical periods like open enrollment for benefits, increasing the likelihood of successful breaches.

Impact and Consequences

The sophistication of these attacks has severe implications, leading to ransomware incidents and catastrophic disruptions for organizations, including hospitals and large corporations. The urgency to combat these threats has prompted IT departments to implement more aggressive phishing simulations.

---

3. IT Departments' Response: Phishing Tests Turn Dire

Aggressive Phishing Simulations

To bolster defenses against real phishing attempts, IT departments across companies and universities have resorted to deceptive testing methods. These simulations often involve sending realistic and alarming fake emails to employees and students to assess their vulnerability.

Notable Examples of Phishing Tests

At [02:06], McMillan shares some striking instances of these tests:

"There was one email, it was about a lost puppy dog in a parking lot... The craziest example that I heard of was the University of California Santa Cruz, which last summer sent a phishing email test themed Ebola outbreak on campus."

These tests aim to simulate the pressure and deceit inherent in real phishing attacks, preparing individuals to respond appropriately.

Educational Approaches and Their Effectiveness

Bienname queries the efficacy of these tests in fostering awareness and resilience. McMillan references alternative educational strategies:

"It's the idea of embarrassing people and putting them in an adversarial position... having phishing awareness months and fun, less shameful kinds of ways of teaching people to report phishing emails and to spot them." ([02:56]).

A study by the University of California, San Diego revealed that traditional phishing education methods yielded minimal improvements—only about a 2% increase in phishing avoidance—suggesting the need for more innovative training approaches.

---

4. Artificial Intelligence and the Challenge of Hallucinations

Understanding AI Hallucinations

AI systems, particularly large language models, sometimes produce erroneous or fabricated information—a phenomenon known as "hallucination." These inaccuracies undermine the reliability of AI applications in critical sectors.

Amazon's Automated Reasoning Initiative

At [05:40], the discussion shifts to Amazon Web Services' (AWS) venture into mitigating AI hallucinations through automated reasoning. Bell Lin explains:

"Automated reasoning is actually a branch of AI... it's using computers to automate the mathematical logic behind putting rules into AI and sort of hard coding it." ([06:15]).

Unlike machine learning, which learns from vast datasets, automated reasoning imposes strict logical frameworks to ensure AI outputs adhere to predefined rules, enhancing accuracy in sensitive applications.

Practical Applications and Industry Impact

Bell Lin highlights Amazon's collaboration with PricewaterhouseCoopers (PwC) as a case study:

"PwC... is using it... ensuring that the regulations are met for how these drugs are marketed." ([07:49]).

This approach is crucial for industries where compliance and precision are non-negotiable, such as pharmaceuticals and finance.

Limitations and the Necessity of Human Oversight

Despite advancements, Bell Lin is cautious about automated reasoning's ability to completely eliminate hallucinations:

"The answer is undecidable. ...it's probably a no..." ([08:55]).

She emphasizes that current solutions can reduce but not fully eradicate inaccuracies, underscoring the continued need for human oversight in verifying AI-generated content.

---

5. Future Outlook and Expert Opinions

Evolving Solutions to AI Challenges

Amazon, along with competitors like Microsoft and Google, is exploring various strategies to mitigate AI hallucinations. These include:

• Retrieval-Augmented Generation (RAG): Enhancing AI responses by retrieving relevant information from verified sources.
• Encouraging AI to Admit Uncertainty: Teaching chatbots to acknowledge when they lack sufficient information rather than providing potentially false answers.

Balancing Creativity and Accuracy

Interestingly, Bell Lin notes that hallucinations are not entirely negative:

"There are actually really great uses for hallucinations in creative sectors... you want a really wacky image because you are a painter..." ([09:24]).

This duality presents a nuanced challenge: harnessing AI's creative potential while ensuring reliability in applications where accuracy is paramount.

The Human Element Remains Crucial

Ultimately, current AI technologies supplement rather than replace human judgment:

"None of the big tech companies are saying that humans should be out of the loop altogether..." ([10:08]).

Professionals across various fields continue to play an essential role in overseeing and validating AI outputs to prevent errors and maintain compliance.

---

6. Conclusion

The episode underscores a critical intersection between cybersecurity and AI reliability. As phishing attacks grow more sophisticated, organizations must evolve their defensive strategies beyond traditional methods. Concurrently, while AI advancements like automated reasoning offer promising avenues to reduce errors, the inherent complexity of these systems ensures that human oversight remains indispensable. The continuous dialogue between technology and human expertise is essential for navigating the challenges and harnessing the benefits of modern innovations.

---

Notable Quotes:

• Bob McMillan at [01:29]:
  "They try to play in your mind. They try to get you in some kind of panic mode. So usually what happens with these phishing emails is there's some very, very important piece of information they promise..."

• Bob McMillan at [04:03]:
  "The hackers are getting very clever. They know how corporations work and they know which kind of emails are high priority..."

• Bell Lin at [06:15]:
  "Automated reasoning is actually a branch of AI... it's using computers to automate the mathematical logic behind putting rules into AI and sort of hard coding it."

• Bell Lin at [08:55]:
  "The answer is undecidable... it's probably a no."

---

Note: The episode also includes promotional segments from Oracle and NetSuite, which have been excluded from this summary to focus solely on the content-driven discussions.

Transcript

[Oracle Representative]

ADP knows any big thing, any small thing, any trendy thing, even a trendy thing that everyone knows isn't a great idea, but management just wants us to give it a try for a bit. Can change the world of work. From HR to payroll, ADP designs forward thinking solutions to take on the next anything.

[Pierre Bienname]

Welcome to Tech News briefing. It's Monday, February 10th. I'm Pierre Bienname for the Wall Street Journal. How useful can artificial intelligence really be if it sometimes makes stuff up? Amazon is turning to so called AUD automated reasoning to cut down on hallucinations. And IT departments routinely try to fool their employees in order to get them to recognize hackers. Phishing attempts and some say they've gone a little too far. Hackers who engage in phishing, sending deceptive emails aimed at stealing sensitive information, are cooking up some increasingly sophisticated scams. As a result, IT departments at companies and universities are throwing sensational tests at their employees and students. The idea is if you opened this email and clicked on the link, you've failed the test. And failure comes at a cost. Phishing, spelled with a ph, was the first step in about 14% of cyberattacks last year. That's according to an analysis of data breaches done by Verizon. Bob McMillan writes about computer security for the Wall Street Journal and he reported on the value of these test traps. So, Bob, how does phishing typically work?

[Bob McMillan]

Well, they try to play in your mind. They try to get you in some kind of panic mode. So usually what happens with these phishing emails is there's some very, very important piece of information they promise, like your vacation days are being cut. And you're like, what? My vacation days are being cut? You click on the link, then you have to log in. You don't even realize you're not on your corporate website. You're on some fake website the hackers set up. So you're giving them information right there.

[Pierre Bienname]

What are some of the examples of these kind of phishing traps set up by IT departments to lure people and maybe trick them into falling for it?

[Bob McMillan]

There was one email, it was about a lost puppy dog in a parking lot. There was a guy who sent an email to NASA staffers promising, you know, a chance to win a ticket to see the final space shuttle launch. And he apparently made a staffer there cry when she realized it was a fake. The craziest example that I heard of was the University of California Santa Cruz, which last summer sent a phishing email test themed Ebola outbreak on campus. And it basically sent some People into a panic there thinking that there was a case of Ebola on campus. There wasn't. They were just trying to do a phishing education test.

[Pierre Bienname]

What are IT departments learning as far as the most effective way to spread awareness and boost resilience against phishing, which is the whole idea.

[Bob McMillan]

I did interview a guy from Google who was talking about some other approaches to curbing this. Like education is not a bad thing. It's the idea of embarrassing people and putting them in an adversarial position and then going like, now listen to me, there are other ways of doing education, having phishing awareness months and fun, less shameful kinds of ways of teaching people to report phishing emails and to spot them. There's some research from the University of California in San Diego that basically looked at a variety of phishing email tests and then educational responses to them. And they found that basically the sort of classical approach to doing this yields negligible results at best. They found sort of a 2% improvement in the, in the likelihood of the targets to avoid phishing emails in the future.

[Pierre Bienname]

Now the phishing attempts of yore look pretty ridiculous. Things like free prizes, I love you in all caps or you know, the notorious email from a Nigerian prince. But what do really advanced phishing attempts look like now?

[Bob McMillan]

The hackers are getting very clever. They know how corporations work and they know which kind of emails are high priority. They, they know like an email from the CEO demanding some kind of immediate response about corporate facts or what's going on with this pitch or something like that. They know those kind of emails are very successful at the open enrollment time of year, you know, when you're re upping your medical. They know that an email with that theme that's sent around November, you know, gets a very high response rate. The problem is these phishing attacks, they lead to ransomware. They lead to like catastrophic consequences for some corporations, for some hospitals, and there's a sense of urgency around stopping them from working.

[Pierre Bienname]

That was WSJ reporter Bob McMillan coming up. AI bots occasionally say the darndest things, giving flat out wrong answers. We hear about the obscure field of research that could help solve that problem. That's after the break.

[Oracle Representative]

AI requires a lot of compute power and the cost for your AI workloads can spiral. That is unless you're running on oci. Oracle Cloud infrastructure. This was the cloud built for AI, a blazing fast, enterprise, enterprise grade platform for your infrastructure, database, apps and all of your AI workloads. Right now Oracle can cut your current cloud bill in Half if you move to OCI. Minimum financial commitment and other terms apply. Offer ends March 31. See if you [e-mail] wallstreet oracle.com Wall Street.

[Pierre Bienname]

Artificial intelligence is known to sometimes make up answers and to share these so called hallucinations with confidence. Now, Amazon's cloud computing unit, Amazon Web Services, is looking to automated reasoning for hard mathematical proof that these errors can be stopped, at least in certain areas. Some analysts say that success could mean millions of dollars worth of AI deals with businesses. Bel Lynn writes about AI and enterprise technology for the Wall Street Journal, and she joins me now. Okay, so Bell, how does this automated reasoning work, this mathematical concept that Amazon is turning to to solve hallucinations?

[Bell Lin]

In part, automated reasoning is actually a branch of AI. So in some ways you can think of it as using AI and math to sort of fight back against a different form of AIs hallucinations or propensity to spit back this inaccurate data. And automated reasoning is really using computers to automate the mathematical logic behind putting rules into AI and sort of hard coding it. So machine learning differs from automated reasoning in that it basically hoovers up a bunch of data and that can be structured or unstructured. Data can be words or text, it could be numbers. And it teaches machines or computers how to capture patterns from that data. So how to separate a dog from a cat, how to identify a number from a letter. And so that's how the machine captures or gets its intelligence. Whereas in automated reasoning, you're sort of hard coding a set of rules and logic into a system.

[Pierre Bienname]

So the AI is able to check itself for errors in a way.

[Bell Lin]

Yeah, that's right. So similar to the way that we've heard that some large language models can reason through problems. Is the system working the way that it's intended? Is the model spitting out an answer that's accurate based on a preset defined set of rules? And those rules for a company can be a set of internal company guidelines for employees, or it could be a product guidebook for customers to know what sorts of services and products you have in your catalog.

[Pierre Bienname]

Okay, so speaking of customers, has Amazon had much success taking this approach to market?

[Bell Lin]

It's relatively new and it's something that they call in preview. So they're certainly testing it and hoping that it really unlocks a lot of business deals for them. Because right now, hallucinations are a big blocker, not just for consumers like you and I to use chatbots more fully in our daily lives, but for businesses who need that reliability when they're doing things like creating advertisements for pharmaceuticals, and they can't run afoul of regulations or at worst, promote something that is completely inaccurate. So, for instance, PricewaterhouseCoopers, the big audit, accounting and tax firm, is actually a customer of Amazon's. And so that's important because on one hand, the AI may be trying to help PricewaterhouseCoopers achieve the goal of creating really good advertising, but that runs up against the goal of ensuring that the regulations are met for how these drugs are marketed. And so you need automated reasoning to come in and say, yes, we are adhering to the regulations, or no, we're not adhering to the regulations.

[Pierre Bienname]

What are experts saying about automated reasoning? Is it something that's really going to put an end to hallucinations?

[Bell Lin]

Oh, absolutely not. When you pose this question to an automated reasoning system, the AWS scientist who told me about how they're using these systems said the answer is undecidable. And so that's a really interesting answer that I interpreted as no, because when automated reasoning can tell you something with accuracy 100% of the time, that means it's probably a no, because it can't say that it's 100% a yes.

[Pierre Bienname]

Are there some other possible solutions?

[Bell Lin]

The solution that Amazon is pushing and its competitors like Microsoft and Google also have something similar for reducing chatbot hallucinations. And so they're saying that maybe the hallucinations can be mitigated or chatbots might be taught to say, I don't know, rather than eliminating them altogether. There are actually really great uses for hallucinations in creative sectors and fields where you want a really wacky image because you are a painter or you want some out of the box song lyric because you're a lyricist. So it's by design that these chatbots hallucinate, but we really do want them to not hallucinate in circumstances where it really, really matters.

[Pierre Bienname]

But until then, it seems like you'll always maybe want a human in the chain to check that things are right.

[Bell Lin]

Yeah, that's right. None of the big tech companies are saying that humans should be out of the loop altogether, that automated reasoning and other methods like retrieval, augmented generation should supplant the need for a human to basically check the output of a chatbot, or for a doctor to check the output of a medical question you input into the system. Price for WaterhouseCooper still has their legal team review the advertisements, for instance. So the chatbot and automated reasoning forms the first layer of checking.

[Pierre Bienname]

That was our reporter Bell Lin. And that's it for Tech News Briefing. Today's show was produced by Julie Chang with supervising producer Kathryn Millsop. I'm Pierre Bienname for the Wall Street Journal. We'll be back this afternoon with TNB Tech Minute. Thanks for listening.

[Oracle Representative]

Okay, business leaders, are you here to play or are you playing to win? If you're in it to win, meet your next mvp. Netsuite by Oracle netsuite is your full business management system in one convenient suite. With NetSuite, you're running your accounting, your finance, your HR, your e commerce, and more, all from your online dashboard. Upgrade your playbook and make the switch to NetSuite, the number one cloud ERP. Get the CFO's guide to AI and machine learning at netsuite.com Wall street netsuite.com Wall Street.