Transcript
Phil Agnew (0:00)
I'm yet to speak to a marketer who thinks AI is better at them than their job. But I wondered if marketers can actually spot the difference between AI generated copy and human created work. So I'm here at Inbound Conference, one of the world's biggest marketing conferences, where over 11,000 marketers attend. My plan is to interview these incredibly smart marketers and find out will they spot my AI generated copy or not? But first, let's rewind to a week ago. Here's how I created this AI copy. So you want to be a marketer? It's easy. You just have to score a ton of leads and figure out a way to turn them all into customers. Plus manage a dozen channels, write a million blogs, and launch a hundred campaigns all at once. When that's done, simply make your socials go viral and bring in record profits. No sweat. Okay, fine, it's a lot of Sweat. But with HubSpot's AI powered marketing tools, launching benchmark breaking campaigns is easier than ever. Get started@HubSpot.com marketers if I told you that this entire podcast episode was generated using AI, would it change your perception of the podcast? Well, I can promise you this episode isn't created by AI. But there's a study that did analyze the perception of AI authored content. The researchers, Angelica Hennes Trosser, Hannah Grieving and Joachim Kimmilar, did so in their 2022 study. Specific researchers wanted to find out how people perceive news articles that AI supposedly writes. To do so, they set up three simple experiments. The participants were recruited online and asked to read news articles. All were told that they were evaluating the quality of science journalism. They were shown articles on 2 Wolves in Germany and autonomous cars. Here, the researchers introduced a twist. The same article was shown to different groups. Some people were truthfully told that the article was written by a human journalist along with the journalist's name and job title, while others were told it was written by an AI system called Automated txt. In reality, all of the articles were written by humans. So what did the 715 participants think of the supposed AI articles? Did they rate them differently from the human ones? Well, the study found something rather surprising. When people were directly asked to compare the AI in human authors, they believed AI was more objective and more accurate, a belief the researchers called the machine heuristic. The participants also said the AI generated articles were more error free and unbiased, and this showed a clear bias among participants for AI generated content. This finding surprised me. It contradicted my previous belief that Readers would critique AI generated content more harshly than work authored by humans if they knew its origin. But this is just one study. It's just looking at news articles. Is there any evidence that AI content is perceived as higher quality across the board? Well, yes, there is. See, this 2022 study I've been talking about on news articles was part of a meta analysis involving another 120 studies with 53,000 participants. The researchers behind this meta analysis studied how persuasive AI is making people believe something like something and decide to do something. How persuasive is AI in these things? Well, the meta analysis revealed that AI is just as effective as humans in persuasion. There is no significant difference overall in how persuasive human text is compared to AI generated text. That said, AI is slightly worse at persuading someone to buy or change behaviours, but is no worse at persuading them to change perceptions or attitudes. Interestingly, AI is more effective in one way communication, but it backfires and can feel less human and less persuasive when there's a back and forth conversation. Surprisingly, it didn't matter much where the people were from individualistic countries like the US or group focused cultures like China. Both were persuaded by AI. And age and gender didn't seem to have a significant effect either, except that millennials so people age aged 25 to 40 were more skeptical of AI. This study was the first of its kind. The meta analysis concluded that AI is generally as persuasive as humans. Now this raises some questions for marketers. If AI is as persuasive as us, can it do a marketer's job? Could it write copy, design websites and draft blogs to convert customers? Many marketers believe it can. Influencer Marketing Hub surveyed 1,290 marketers and found that 70.6 believe AI can outperform humans in key marketing tasks, while nearly 60% fear it could replace their jobs. The fear of AI jeopardizing jobs has actually doubled since 2023, up from 35% back then. News articles written by AI are seen as less objective and more error free. A meta analysis found that AI is as persuasive at changing attitudes and perceptions, and the majority of marketers are now fearful about their jobs, worrying AI will take over. But is this all a little overhyped? Can AI generated copy really outperform the best marketers? Well, I wanted to find out myself with my own test. In a few weeks I'll be heading to HubSpot's Inbound conference. Inbound is one of the world's largest marketing conferences and 12,000 of the world's largest, world's smartest marketers are attending. I plan to show AI generated copy and human written copy to some of the copywriters, digital marketers, brand managers and CEOs attending that event. I've picked four main sponsors at the Inbound Webflow, Rollworks, SuperD, and Wistia. I've opened up their websites and I've taken their top of the fold Copy this is the line of text that appears on the header of the website as soon as you open the page. Arguably it is the most or it should be the most persuas line of copy on the site. It anchors the visitor's entire experience. And in my experience as a marketer at large tech companies, this is the line of copy that marketers spend the most time working on. So I took that line of copy And I asked ChatGPT to come up with something better. Here's the instruction I gave ChatGPT. I'd like you to help me with the title copy of this company's website. Here's more information about the company. So here I added a line of information. It was tailored for each company. For example, with webflow's visual first platform, marketers, designers and devs alike can create, optimize and scale web experiences that get results. All backed by enterprise grade hosting and security. And here's the original title copy More than a Website Builder, Please improve it while keeping the length the same for Webflow. ChatGPT came up with this copy beyond a website builder. It changed roleworks tagline from account based execution made easy to account based strategy simplified it changed SuperD's copy from turn sales reps into CRM superheroes to transform sales reps into CRM champions. And it altered Wistia's copy from video for business is our business to Business Video solutions our expertise. So will marketers spot my AI fakery? Well find out after this short break. Create like the Greats, hosted by Ross Simmons, is brought to you by the HubSpot Podcast Network, the go to audio destination for business professionals. In each episode, Ross dives into the stories behind some of history's greatest creations and creators. He unpacks the strategies, processes, and lessons that shaped them. His episodes are engaging, his insights are practical, and he's been living these principles that he shares for over a decade. If you enjoy exploring creativity, the history of creators, and actionable advice, this podcast is for you. Listen to Create like the Greats wherever you get your podcasts. Hello and welcome back to Nudge. To test my AI generated marketing copy, I found 12 experienced marketers at the inbound conference. Here they are. Carlos Gomez.