Loading summary
Phil Agnew
I'm yet to speak to a marketer who thinks AI is better at them than their job. But I wondered if marketers can actually spot the difference between AI generated copy and human created work. So I'm here at Inbound Conference, one of the world's biggest marketing conferences, where over 11,000 marketers attend. My plan is to interview these incredibly smart marketers and find out will they spot my AI generated copy or not? But first, let's rewind to a week ago. Here's how I created this AI copy. So you want to be a marketer? It's easy. You just have to score a ton of leads and figure out a way to turn them all into customers. Plus manage a dozen channels, write a million blogs, and launch a hundred campaigns all at once. When that's done, simply make your socials go viral and bring in record profits. No sweat. Okay, fine, it's a lot of Sweat. But with HubSpot's AI powered marketing tools, launching benchmark breaking campaigns is easier than ever. Get started@HubSpot.com marketers if I told you that this entire podcast episode was generated using AI, would it change your perception of the podcast? Well, I can promise you this episode isn't created by AI. But there's a study that did analyze the perception of AI authored content. The researchers, Angelica Hennes Trosser, Hannah Grieving and Joachim Kimmilar, did so in their 2022 study. Specific researchers wanted to find out how people perceive news articles that AI supposedly writes. To do so, they set up three simple experiments. The participants were recruited online and asked to read news articles. All were told that they were evaluating the quality of science journalism. They were shown articles on 2 Wolves in Germany and autonomous cars. Here, the researchers introduced a twist. The same article was shown to different groups. Some people were truthfully told that the article was written by a human journalist along with the journalist's name and job title, while others were told it was written by an AI system called Automated txt. In reality, all of the articles were written by humans. So what did the 715 participants think of the supposed AI articles? Did they rate them differently from the human ones? Well, the study found something rather surprising. When people were directly asked to compare the AI in human authors, they believed AI was more objective and more accurate, a belief the researchers called the machine heuristic. The participants also said the AI generated articles were more error free and unbiased, and this showed a clear bias among participants for AI generated content. This finding surprised me. It contradicted my previous belief that Readers would critique AI generated content more harshly than work authored by humans if they knew its origin. But this is just one study. It's just looking at news articles. Is there any evidence that AI content is perceived as higher quality across the board? Well, yes, there is. See, this 2022 study I've been talking about on news articles was part of a meta analysis involving another 120 studies with 53,000 participants. The researchers behind this meta analysis studied how persuasive AI is making people believe something like something and decide to do something. How persuasive is AI in these things? Well, the meta analysis revealed that AI is just as effective as humans in persuasion. There is no significant difference overall in how persuasive human text is compared to AI generated text. That said, AI is slightly worse at persuading someone to buy or change behaviours, but is no worse at persuading them to change perceptions or attitudes. Interestingly, AI is more effective in one way communication, but it backfires and can feel less human and less persuasive when there's a back and forth conversation. Surprisingly, it didn't matter much where the people were from individualistic countries like the US or group focused cultures like China. Both were persuaded by AI. And age and gender didn't seem to have a significant effect either, except that millennials so people age aged 25 to 40 were more skeptical of AI. This study was the first of its kind. The meta analysis concluded that AI is generally as persuasive as humans. Now this raises some questions for marketers. If AI is as persuasive as us, can it do a marketer's job? Could it write copy, design websites and draft blogs to convert customers? Many marketers believe it can. Influencer Marketing Hub surveyed 1,290 marketers and found that 70.6 believe AI can outperform humans in key marketing tasks, while nearly 60% fear it could replace their jobs. The fear of AI jeopardizing jobs has actually doubled since 2023, up from 35% back then. News articles written by AI are seen as less objective and more error free. A meta analysis found that AI is as persuasive at changing attitudes and perceptions, and the majority of marketers are now fearful about their jobs, worrying AI will take over. But is this all a little overhyped? Can AI generated copy really outperform the best marketers? Well, I wanted to find out myself with my own test. In a few weeks I'll be heading to HubSpot's Inbound conference. Inbound is one of the world's largest marketing conferences and 12,000 of the world's largest, world's smartest marketers are attending. I plan to show AI generated copy and human written copy to some of the copywriters, digital marketers, brand managers and CEOs attending that event. I've picked four main sponsors at the Inbound Webflow, Rollworks, SuperD, and Wistia. I've opened up their websites and I've taken their top of the fold Copy this is the line of text that appears on the header of the website as soon as you open the page. Arguably it is the most or it should be the most persuas line of copy on the site. It anchors the visitor's entire experience. And in my experience as a marketer at large tech companies, this is the line of copy that marketers spend the most time working on. So I took that line of copy And I asked ChatGPT to come up with something better. Here's the instruction I gave ChatGPT. I'd like you to help me with the title copy of this company's website. Here's more information about the company. So here I added a line of information. It was tailored for each company. For example, with webflow's visual first platform, marketers, designers and devs alike can create, optimize and scale web experiences that get results. All backed by enterprise grade hosting and security. And here's the original title copy More than a Website Builder, Please improve it while keeping the length the same for Webflow. ChatGPT came up with this copy beyond a website builder. It changed roleworks tagline from account based execution made easy to account based strategy simplified it changed SuperD's copy from turn sales reps into CRM superheroes to transform sales reps into CRM champions. And it altered Wistia's copy from video for business is our business to Business Video solutions our expertise. So will marketers spot my AI fakery? Well find out after this short break. Create like the Greats, hosted by Ross Simmons, is brought to you by the HubSpot Podcast Network, the go to audio destination for business professionals. In each episode, Ross dives into the stories behind some of history's greatest creations and creators. He unpacks the strategies, processes, and lessons that shaped them. His episodes are engaging, his insights are practical, and he's been living these principles that he shares for over a decade. If you enjoy exploring creativity, the history of creators, and actionable advice, this podcast is for you. Listen to Create like the Greats wherever you get your podcasts. Hello and welcome back to Nudge. To test my AI generated marketing copy, I found 12 experienced marketers at the inbound conference. Here they are. Carlos Gomez.
Simon
I'm Simon and I'm an account executive at HubSpot. Chris Savage, CEO of Wistia.
Phil Agnew
My name is John Weigel. I'm a podcast host. My name is Lindsey Gambo. I'm Allison. And I'm Maggie. My name's Drew. My name is Jess. My name is Matt Warner. My name is Meredith Jacobson. My name is Leanne. Before my test, I asked this talented bunch a simple question. First question, do you think ChatGPT can write better copy to marketers?
Simon
No, but it gives a good starting point and probably needs to be tweaked.
Phil Agnew
No, I do not. Yes, to a certain extent.
Simon
No.
Phil Agnew
No, I don't think so. I mean, it's missing the personalized angle, the tone of voice, the kind of.
Simon
The messaging that you're trying to get across.
Phil Agnew
I think ChatGPT can help marketers organize their thoughts, but I don't think it can replace human creativity and opinion. These experienced marketers seem pretty bullish. Most don't think that ChatGPT can write better copy than marketers. But let's put that to the test. What I've done is I've taken four of the main sponsors here. Ooh, exciting. Okay. I've gone to their website about a month ago, looked at their copy above the fold. You know the headline copy? Yeah. Copied and pasted it, put it in Chat GPT to make it better. This is so exciting. Okay.
Simon
Yeah.
Phil Agnew
Versions are quite similar, so don't, you know, you might not get all of them. All right, hang on. See how many you can correctly say is AI. Okay, you ready for the first one? Yeah, let's do it. Webflow. All right. Okay. Option one, beyond a website builder. Option two, more than a website builder. Which one is it? I'm gonna go. So I have to guess which one was AI. Right. Let's go beyond a website builder. I think the beyond is the AI. I feel like AI was beyond. Beyond a website builder is AI. Yes, that is correct. The marketers were very accurate here. The majority correctly noticed that beyond a website builder was AI generated. But let's face it, there's a 50, 50 chance they would get that one right. So let's try another company. Number two, Role works. Option one, Account based strategy simplified. Option two, account based execution made easy. Which is AI. The simplified one is AI. Yeah. Correct. Option two. Option two is AI. Yes, that's incorrect. Oh, boy. Second One. Second one's AI. Yeah. Correct. Simplified is AI. You think simplified is AI? I think option one is AI.
Simon
It's a satorin.
Phil Agnew
Simplified as AI. Yeah, I like number two better. So number one must be AI. Confident as well. You're both wrong. Number two is AI.
Simon
I would go option one being AI.
Phil Agnew
Option one's AI. That's incorrect.
Simon
Ah, here we go.
Phil Agnew
The marketers fared a bit worse here. Half of the marketers I spoke to got this one wrong. All right, third one, SuperD. Option one, transform sales reps into CRM champions. Option two, turn sales reps into CRM superheroes. Oh, which one's AI? Oh, I'm gonna say.
Simon
Number two is AI.
Phil Agnew
Incorrect again.
Simon
Ah, I was thinking the super and the supers. The first one is AI.
Phil Agnew
Carlos, that's incorrect. Okay. Number two. That's correct. Champion is a weird word to use. Option two, AI. Yes. Correct. I think champions has to be AI. I agree. Correct. Three for three. I think the superhero wants the AI. That's incorrect. No, again, despite their confidence at the start, many marketers I spoke to failed to spot this AI Generated copy. But let's see how they fared on the last question. This company is Wistia. Option one, Video for business is our business. Option two, Business Video Solutions, our expertise.
Simon
I think the second one's AI.
Phil Agnew
Second one's AI. Yeah, that is correct.
Simon
Option two.
Phil Agnew
Correct. Yeah.
Simon
You gotta say, I think the second one.
Phil Agnew
It's the second one. I. It has to be the first one. It is. Okay, well done, guys. So you're both going option two. Yeah. You're both correct. I want to go with option two. AI. Yeah, correct. Let's go number one as AI. Yeah, Bang on that. Oh, yeah. Three and one.
Simon
Surely number two is AI.
Phil Agnew
Surely. Has to be.
Simon
Has to be.
Phil Agnew
You are correct. Thank God.
Simon
The second one is AI.
Phil Agnew
The second one is AI. Yeah. Video for business is our business. Yeah, Carlos. That's the original. Almost all of the marketers other than Carlos got this one right, including, perhaps unsurprisingly, the Wistia CEO. Business Video Solutions, our expertise.
Simon
The first one.
Phil Agnew
First one's correct. Thank God. And Gina, you just knew, didn't you? You knew.
Simon
Well, no, the other one didn't sound like something we would write.
Phil Agnew
Overall, the marketers were fairly good at spotting AI generated copy. On average, 68% of them noticed when ChatGPT had crafted the copy. So to find out how senior marketers feel about AI and its implications, I spoke to Richard Trunkelli, the senior manager of marketing operations, live At Inbound.
Simon
My name is Richard Troncali and I'm with Rakuten Advertising. I'm our senior manager of marketing operations over the last three years for the company.
Phil Agnew
One of the things that I really want to talk about today, as you've hinted on, is how AI is progressing, how it's changing, how it's affecting our jobs as marketers. You're a bit of a veteran marketer. You've been doing this for a long time. You've been in the game for a while. How have you seen AI change over the years? How have you seen it evolve? What's your take on it?
Simon
Yeah, when I did social media 10 years ago and eight years ago, we used to have to really think and sit in the room and really think, what's our post going to be for the day? What are we going to do for the next few days? And I look back at social media, really how AI and how it transformed. It's exciting to see the next wave of what we're doing with this tool. And from, from a content standpoint, AI really focuses on really enhancing the conversation, but doesn't take it away altogether.
Phil Agnew
Totally agree. It definitely helps. But one of the fears that I have, and I think the fears that a lot of the marketers I've spoken to here at Inbound have is how is AI going to evolve and could it eventually get to a state where it could be replacing some of our jobs?
Simon
Not to scare people, but I think there is a chance that there will be a huge transformation. And if you're not on the bandwagon of it and you're really not focusing on evolving to AI, there's a chance you may lose your job. But for us, it went from learning your skills before using these types of tools to now really focusing on how can I use AI to improve what I do and be an asset to my company? And for me, I really feel that with AI you just have to enhance and you have to be excited about it, learn it. And if you don't, that's where you may start to lose your skills with marketing. So, yeah. So I really think it depends on how you really interpret AI.
Phil Agnew
So you're 50, 50. I feel 50, 50 as well. I think one of the reasons that I'm worried about AI is because I feel like if my audience realizes I'm using AI and if they spot that AI content, they'll value my content less. They'll think I'm putting less effort into it. They'll see the value less. Do you think that customers, if they notice that a marketer is using AI, do you think they would care?
Simon
If I take the side of myself with my company and just think myself as a consumer, I care. I care a lot. I'm constantly reading different things, whether it's learning about more stuff in my field or reading an article about some sport team that I like. And you can really tell the difference when you're reading something that has an AI generated piece of content versus somebody writing it. There's that emotional connection that you feel when you're reading something that somebody wrote.
Phil Agnew
Yeah, totally agree. Thinking back to the quiz, a lot of the. A lot of them got that role. The role works one wrong. A lot of people got other ones right. I've sort of got this hypothesis that really strong positioning, really strong strategy. A company that has very clear positioning maybe is making a few sacrifices in their positioning as well. Like saying, we are not for X, we are for Y. That seems to be the kernel that allows you to create copy that AI can't copy. How important is positioning and strategy when it comes to marketers to future proof their jobs and create great copy?
Simon
Absolutely. It's interesting. So in terms of your global strategy and how you're using AI and how you're just really positioning yourself as a brand, you need to make sure that you're not solely relying on AI. If you use AI for every single piece of content, everything you do, you're gonna get too broad and high level almost. You're not really focusing on your niche. You're not targeting your customers with the emotions that they're wanting. You're giving them something that's robotic. And again, I do like AI and I think it's really good. But I still feel and value that we need to make sure for our brand and for global strategy that there's people behind that executing it and not just a machine.
Phil Agnew
All right, Richard, thank you so much. Now, looking back on my very unscientific experiment, I've got to admit that it was pretty unfair on the marketers. The copy was very short, fairly vague, and quite high level. It's almost impossible to notice the AI version consistently, even for the most tenured marketers. But I found it interesting that the marketers were far more likely to notice the AI version of the Wistia copy. 10 out of the 12 correctly identified the AI fakery here. And I think that's because the Wistia copy is very good. The line video for business is Our business is brilliant in many ways. You have alliteration and repetition and that'll make the copy more believable and memorable. Richard Shotton's study on alliterating versus non alliterating proverbs found that alliteration made the proverbs 7% more believable and 22% percent more memorable. That will be working in Wistia's favour as well. But it's not just alliteration, it's much more than that really. It's smart positioning. Wistia are purposefully ignoring some potential clients with this message. They say video for business is our business, which is the equivalent of saying if you're not a business, don't bother buying from us. It's a pretty strong position from a video hosting company, especially as most of their competitors also offer their software to individual consumers to non business consumers. By focusing the message, Wistia has come up with a unique and trustworthy line. Their message incurs a cost. It rules out some potential prospects and makes the message more potent it reminds me of a quite interesting empirical study on reverse psychology and advertising messages. The study showed pairs of advertisements to 52 university students. Some saw a standard positive ad message like Buy this jacket, while others saw the reverse psychology message. Literally. It said don't buy this jacket. The participants then chose which message they found more interesting. Now, 76% of the participants said they found the reverse psychology uncomfortable, but nearly half were more interested by the reverse psychology ads, with 48% of those who felt uncomfortable still saying they were more interested by it. While these unconventional messages they might cause discomfort, they did spark curiosity. They did capture attention. And Wistia? They're not using reverse psychology. They don't tell people not to buy. But by clearly stating who they're for in their opening copy, I think they're benefiting from the same principle. They are going against the grain. Most marketers and most AIs tasked with creating marketing will appeal to the broadest possible customer base. They believe, incorrectly, that targeting the masses will attract the most sales. But Wistia know that focusing their offer will actually increase their sales. And that type of strategy is something that AI I think will struggle to replicate. I started today's episode wondering if marketers would notice my AI copy. 68% of them did. But the tide is changing. A 2022 study found that AI authored news articles are perceived incorrectly as more objective and error free. A 2023 meta analysis found that AI is as good as humans at persuading customers to change their attitudes and perceptions. And a 2024 survey states that 60% of marketers are worried about AI taking their jobs. However, if you craft copy like Wistias copier that highlights your positioning and is built on the foundation of a strong strategy, you will create marketing that AI simply can't replicate. Who knows what the future holds. But for now, I think it's clear that we marketers still have the edge over AI. However, I do doubt that that advantage will last forever. That is all for today folks. Thank you so much for listening. First of all, a massive thank you to the 12 HubSpot attendees who agreed to take part in my quiz, and to Richard Trunkelly for sharing his thoughts on AI. And of course to HubSpot for inviting me to attend and talk at that fantastic Boston Inbound event. Before ending, I want to remind you about the big change coming to Nudge this year. I am now creating two weekly episodes. Last Friday I launched my first Friday episode. The Friday episodes are shorter, they are guest free and that allows me to dive into very specific topics. The debut episode explored randomized controlled trials and their history, their importance in modern business and I really enjoyed putting it together. This Friday I'm releasing a mini documentary on the Francesca Gino scandal and I think a lot of you will really enjoy that one. But I would love to know what you actually think. So please do get in touch and let me know what you think of these Friday episodes. The easiest way to do that is to connect with me on LinkedIn. Just search Phil Agnew and message me on there. Okay, that is all. Thank you again for listening and you will hear from me this Friday. Cheers.
Podcast Summary: Nudge – Episode: "Can I Fool Veteran Marketers with My AI Fakery?"
Podcast Information:
In the latest episode of Nudge, host Phil Agnew embarks on an intriguing experiment to determine whether seasoned marketers can distinguish between AI-generated copy and human-crafted content. Setting the stage at Inbound Conference—a premier marketing event attracting over 11,000 attendees—Phil aims to uncover the perceptual differences and potential threats AI poses to the marketing profession.
Phil begins by sharing his skepticism about the notion that AI can surpass human marketers. To challenge this, he presents an AI-generated marketing copy created using HubSpot's AI-powered tools. The copy emphasizes the ease of using AI in marketing tasks, subtly questioning whether listeners would perceive the content differently if they knew it was AI-generated.
Notable Quote:
"I'm yet to speak to a marketer who thinks AI is better at them than their job. But I wondered if marketers can actually spot the difference between AI generated copy and human created work."
— Phil Agnew [00:00]
Phil references a 2022 study by Angelica Hennes Trosser, Hannah Grieving, and Joachim Kimmilar, which investigated how readers perceive AI-authored news articles compared to those written by humans. The study involved 715 participants who evaluated articles on topics like wolves in Germany and autonomous cars. Participants were either informed that the articles were human-written or AI-generated.
Key Findings:
Notable Quote:
"When people were directly asked to compare the AI in human authors, they believed AI was more objective and more accurate, a belief the researchers called the machine heuristic."
— Phil Agnew [05:00]
Expanding on the initial study, Phil cites a 2022 meta-analysis encompassing 120 studies with 53,000 participants. This analysis explored AI’s effectiveness in persuasion compared to human-generated content.
Key Insights:
Notable Quote:
"The meta analysis revealed that AI is just as effective as humans in persuasion. There is no significant difference overall in how persuasive human text is compared to AI generated text."
— Phil Agnew [07:00]
Phil discusses insights from a survey by Influencer Marketing Hub, which polled 1,290 marketers. The results revealed that:
Notable Quote:
"A 2022 study found that AI authored news articles are perceived incorrectly as more objective and error free. A 2023 meta analysis found that AI is as good as humans at persuading customers to change their attitudes and perceptions. And a 2024 survey states that 60% of marketers are worried about AI taking their jobs."
— Phil Agnew [17:00]
To test his hypothesis, Phil conducted an experiment at Inbound Conference by presenting AI-generated and human-written copy to 12 experienced marketers from four major sponsors: Webflow, Rollworks, SuperD, and Wistia.
Phil extracted the top-of-the-fold copy from each sponsor's website—the crucial headline that anchors visitor engagement. He then instructed ChatGPT to enhance each headline while maintaining length. The marketers were tasked with identifying which version was AI-generated.
Notable Quote:
"I've taken four of the main sponsors here... put them in Chat GPT to make it better. This is so exciting."
— Phil Agnew [09:33]
The marketers' ability to discern AI-generated copy varied across the sponsors:
Webflow:
Rollworks:
SuperD:
Wistia:
Overall, 68% of the marketers successfully identified the AI-generated copy, indicating a moderately effective detection rate.
Notable Quotes:
"Overall, the marketers were fairly good at spotting AI generated copy. On average, 68% of them noticed when ChatGPT had crafted the copy."
— Phil Agnew [13:56]
"If I told you that this entire podcast episode was generated using AI, would it change your perception of the podcast? Well, I can promise you this episode isn't created by AI."
— Phil Agnew [00:00]
Phil analyzes the results, highlighting why certain AI-generated copies like Wistia's were more easily detected. He attributes this to strategic positioning and clever use of language that AI typically struggles to replicate.
Key Insights:
Phil references a study on reverse psychology in advertising, which found that unconventional messages can spark curiosity despite causing discomfort. This aligns with Wistia's effective copy strategy.
Notable Quote:
"Their message [Wistia] incurs a cost. It rules out some potential prospects and makes the message more potent... this type of strategy is something that AI I think will struggle to replicate."
— Phil Agnew [19:00]
Phil concludes by reflecting on the evolving landscape of AI in marketing. While AI currently matches human marketers in many respects, leveraging strong strategies and unique positioning remains a human advantage. However, he acknowledges that this edge may diminish as AI continues to advance.
Final Thoughts:
Closing Quote:
"If you craft copy like Wistia's copy that highlights your positioning and is built on the foundation of a strong strategy, you will create marketing that AI simply can't replicate. ... I think it's clear that we marketers still have the edge over AI."
— Phil Agnew [19:30]
Phil also announces changes to the Nudge podcast format, introducing shorter, guest-free Friday episodes focusing on specific topics. He invites listeners to connect with him on LinkedIn to share feedback.
Conclusion
This episode of Nudge provides a compelling exploration of AI's capabilities and limitations in the marketing domain. Through scientific studies and practical experiments, Phil Agnew demonstrates that while AI is a powerful tool, human creativity, strategic positioning, and emotional connection remain pivotal in effective marketing. The insights gleaned underscore the importance of integrating AI thoughtfully to complement, rather than replace, human expertise in the ever-evolving marketing landscape.