How to build an AI-generated focus group

Summary

---

The Digiday Podcast

Episode: How to build an AI-generated focus group
Date: November 4, 2025
Host(s): Kamiko McCoy & Tim Peterson
Guest: Tracy Averbon, GM, The Times & The Sunday Times

---

Episode Overview

This episode explores how media organizations are leveraging AI to create "synthetic research panels"—essentially, AI-generated focus groups trained on real audience data. Tracy Averbon, General Manager at The Times and The Sunday Times, discusses their approach, the technology partners involved, validation methods, and key challenges in balancing innovation with editorial integrity. The episode also touches on major media business developments like YouTube vs Disney carriage disputes, potential acquisitions in the streaming industry, and the surge of short-form video as a revenue driver.

---

Key Discussion Points and Insights

1. Industry Climate: Publishers, AI, & Revenue Pressures

• Tim returns from the Digiday Publishing Summit Europe, highlighting pervasive publisher anxieties around AI eroding referral traffic.
• Increasing marketing orientation at publisher events, with publishers using AI-driven ad tools (Google Performance Max, Meta Advantage+).
• “It was almost like everyone kind of just needed to get that stuff off their chest… AI is eating every publisher's lunch. What do you do about it?” — Tim Peterson [02:00]
• “Publishers need to be spending on paid search ads, they need to be spending on Meta… Costs are never great for media companies.” — Tim Peterson [02:30]

2. Synthetic Research Panels: The Times’ Approach to AI Focus Groups

What is a "Synthetic Research Panel"?

• AI models trained on real audience and behavioral data to mimic responses of target audiences for feedback, focus group, and product development purposes.
• Tracy Averbon prefers the term "synthetic research" over “synthetic audiences”.
• Allows quick, scalable, and often accurate audience insights without running full human panels.

Implementation & Validation at The Times

Development Timeline & Principles

• “We’ve been working on this for about eight, nine months... It’s not something you just switch on.” — Tracy Averbon [28:16]
• Built on a rich base of 642,000 paying subscribers with deep data on behavior and preferences.
• The challenge: The Times understands subscriber needs better than prospective/non-subscriber audiences; AI helps bridge this gap.

Partner Selection: Electric Twin

• Selected Electric Twin for incorporating behavioral and social science academic research atop typical LLM and market data. [30:35]
• "The difference... was that layer of behavioral data." — Tracy Averbon [30:35]

Accuracy & Trust in Synthetic Panels

• 10% “holdout” validation method: real panel vs. synthetic responses.
• Achieved ~92% accuracy (general research typically ~93%, due to human inconsistency).
• “General research is only 93% accurate. This is 92% accurate.” — Tracy Averbon [33:08]

Case Example: Naming a Podcast

• Both real and synthetic panels validated the name “The Business” for a new podcast, which subsequently did very well at launch.
• “We gave [the synthetic panel] about four different names… ‘The Business’ came really, really high with the demographic that we were really trying to target.” — Tracy Averbon [33:57]

---

3. Balancing Innovation with Editorial Integrity & Audience Trust

Varied Audience Attitudes Toward AI

• Loyal subscribers: Skeptical about AI, value human curation, wary of overpersonalization.
• Prospective/"news reading" audience: Prefer AI-powered features—summaries, explainers, more video, instant gratification.
• “Our loyal readers are very skeptical about AI… The prospect audiences were very different, they wanted summaries, AI explainers, more video.” — Tracy Averbon [35:56 & 36:00]
• Highlights need for careful balance: innovating for growth while protecting trust and value for core subscribers.
• “What is the level of personalization that's not jarring, that's helpful… but in the same way maintains integrity?” — Tracy Averbon [37:25]

Editorial Concerns and Industry Risks

• AI risks: hallucinations by LLMs, potential misattribution of errors to publishers.
• “A lot of these LLMs are stealing publisher traffic… and the publisher [is] also being blamed when they get it wrong.” — Tracy Averbon [37:54]
• Newer audiences are skeptical of humans (possible editorial bias), while loyalists are skeptical of AI.
• “You have your loyal audience who's skeptical of A.I.; you have the news reading audience… skeptical of humans.” — Host 1 [39:04]

Product Implications

• Not yet implementing AI-generated article summaries, but exploring features to provide deep context and discovery for readers.
• “How do we infuse [AI] in a way that's helpful?” — Tracy Averbon [41:15]

---

4. Synthetic Research in Organization Workflow

Integration & Usage

• Electric Twin embedded with The Times' data and insights teams for months, assuring model quality.
• Panels for both active subscribers and the broader UK news audience.
• SaaS license model: access generally limited to Data & Insights, Marketing teams, not organization-wide to prevent misuse/overreliance.
  • “Otherwise… you can weaponize data to make a point or an argument.” — Tracy Averbon [46:19]

Cost & Accessibility

• Vendors often provide 3-month proofs of concept for free; full SaaS contracts depend on seats/users, not per-query charges.

---

5. Media Industry News & Short-form Video Trends

YouTube vs. Disney Carriage Dispute (06:05–14:45)

• YouTube TV muscling into pay TV, flexing power with Disney/ESPN in deal negotiations.
  • “YouTube TV will become the top pay TV provider in the US in 2026.” — Tim Peterson [06:50]
• YouTube wants shorter deal terms; Disney wants long-term security; biggest loser in the short-term: subscribers facing channel blackouts.
  • “Ultimately, like subscribers are kind of always the losers in this case because you’re still paying for YouTube TV.” — Tim Peterson [13:56]

Netflix & the Warner Bros. Discovery Bidding War (15:17–19:37)

• Reuters reported Netflix is exploring a bid for Warner Bros. Discovery’s streaming and studio assets (not cable).
• Paramount, Amazon, and Apple also rumored as interested, but differences in what each wants from the business.
• “Ultimately that's all it's going to come down to, whoever makes the biggest offer.” — Tim Peterson [18:28]

Short-form Video as a Revenue Engine (20:03–25:29)

• Meta (Reels) $50B revenue run rate (caveated as projection). [21:47]
• YouTube Shorts now generates more revenue per watch hour than long-form video—signals major shift in platform priorities and potential squeeze on long-form creators.
  • “YouTube Shorts now generates more revenue per watch hour than long form videos, which is wild to me.” — Tim Peterson [22:25]
• Implications for "middle-class" creators who focused on YouTube long-form.
• “This puts YouTube long form creators very much in that crunched position.” — Tim Peterson [24:48]

---

Notable Quotes & Memorable Moments

• “The Times is using fake audiences for its marketing—not quite the case, at the same time, kind of the case.” — Host 1 [27:34]
• “We think about [AI audience panels] as synthetic research and a way to get to market quicker, get an answer quicker, take the guesswork out.” — Tracy Averbon [28:16]
• “General research is only 93% accurate. This is 92% accurate.” — Tracy Averbon [33:08]
• “How do we continue to serve our loyal readers… but also bring these new audiences, build products and services for them…without alienating our existing loyal subscriber base?” — Tracy Averbon [36:58]
• “You can weaponize data to make a point or an argument.” — Tracy Averbon [46:19]
• “Now we’re at the point where we’re really focused on our business strategy and then how we can use AI tools to supercharge that.” — Tracy Averbon [46:21]

---

Timestamps for Key Segments

• 00:10–03:20: Publishing Summit Europe recap; AI’s impact on publishers
• 03:20–05:22: Explaining "synthetic audiences"/research panels; The Times’ use case
• 27:27–49:32: In-depth interview with Tracy Averbon on synthetic research: accuracy, implementation, use cases, internal cultural balance, and technology partners
• 06:05–14:45: YouTube vs Disney carriage dispute deep dive
• 15:17–19:37: Potential Warner Bros. Discovery acquisition—industry implications
• 20:03–25:29: Earnings: Short-form video’s rise as a major revenue driver (Meta, YouTube)

---

Tone & Style

The conversation is candid, journalistic, and pragmatic, blending technical explanations with practical business realities and the human impact of change. There’s an undercurrent of skepticism toward hype and a focus on empirical validation.

---

Summary Takeaways

• AI-generated focus groups (“synthetic research panels”) are a practical, increasingly accurate tool for publishers to expedite decision-making and understand new (and existing) audiences without exhaustive surveys.
• Validation and responsible application are essential, both to avoid over-reliance and to preserve audience trust, brand integrity, and internal decision-making rigor.
• Media business models continue to be disrupted—by platform disputes (YouTube vs. Disney), accelerating consolidation (Netflix/Warner), and the dominance of short-form video for monetization.
• Product and editorial teams must balance serving traditional, skeptical subscribers and attracting newer, tech-savvy audiences—with AI both an opportunity and a challenge on all fronts.

---

Transcript

[Host 1]

Foreign.

[Kamiko McCoy]

Hello, hello and welcome to the Digiday Podcast, a show about the business of media and marketing. I'm Kamiko McCoy, senior marketing reporter here at Digiday.

[Tim Peterson]

And I'm Tim Peterson, executive editor of video and audio Digiday Media. What's up, Kamiko?

[Kamiko McCoy]

Welcome back, Tim. We missed you while you were out.

[Tim Peterson]

Thanks. Thanks for having me back.

[Kamiko McCoy]

Having you back. Yes.

[Tim Peterson]

So I was in Portugal for the Digiday Publishing Summit Europe last week. I was there for a week. It took me a week to adjust to the time zone difference, the seven hour time zone difference going from California to Portugal. Now I'm looking at it. It's going to take me a week to adjust the time zone difference coming back the other way. I've never had jet lag like this.

[Kamiko McCoy]

Well, also you've got daylight, daylight savings which is also coming to get. It's looking so fighting on all fronts here.

[Tim Peterson]

Yeah, but, but I mean it was, it was a good time in Portugal. Like, you know, we got a bit of sun. The, the cloudiness maybe was in the town hall room with the publishers talking about AI and referral traffic problems. But it was interesting because like it was almost like everyone kind of just needed to get that stuff off their chest. And then it felt like there was more constructive conversation of like, all right, yeah, AI is eating every publisher's lunch. What do you do about it? And there was more talk of marketing at a publishing summit than I can remember from any publishing summit in terms of like, publishers need to be spending on paid search ads, they need to be spending on Meta at one point during the town hall. And I think Sarah Guaglioni, our senior media reporter for her media briefing coming out this week. So the November 6th edition is going to be recapping some of this. But like there were publishers talking about how they're taking advantage of things like Google's performance Max and Meta's advantage plus like their AI tools for ads in order to get audiences that way, which that's costly. Costs are never great for media companies. On the other hand, it felt like, oh, some publishers are like have ideas of what to do right now other than kind of the panic or the like age old. And I say this as a journalist, news is important. There needs to be a value put on news like as if we all should be compensated just by virtue of existing. Which would be nice. Let me correct this sounds fantastic, but in a world in which people are being stripped of food stamps.

[Kamiko McCoy]

Correct.

[Tim Peterson]

I don't know that we have as strong of an Argument to be like, news is important, give us money.

[Kamiko McCoy]

No, I don't know if the finger wagging is going to work this go round, but you actually had an interesting guest this week, the general manager at the Times in the Sunday Times to talk a little bit about AI, right?

[Host 1]

Yeah, yeah.

[Tim Peterson]

So this was a really interesting conversation. This was, I think, the conversation that I heard the most from people about after the session during the event. So we talked about. Well, the term that we've used in the past on this show is synthetic audiences. The term that sort of Tracy Averbon, who's the GM of the Times and the Sunday Times term she prefers is synthetic research. Whichever term you want to use. Basically, what we're talking about is taking an audience panel, a set of humans and the data associated with them, and plugging them into an AI model to create like an AI avatar or like a simulated version of that audience. So when we had Michael Berge, our senior media buying editor, on the show back in August, and we were talking about, like, agencies and AI, he talked about how some agencies were investing, I think Dentsu is one of them, were investing in synthetic audiences, basically, like AI focus groups. That's what the Times is doing. Their preferred term is synthetic research panels. Again, whether we're calling it synthetic audiences, synthetic research panels, it's a really interesting idea of using AI to run focus groups so that you could, you know, in their case, one of the things that they asked is they were creating a new business news podcast. They weren't sure exactly what they wanted to call it. They would have loved to be able to ask, you know, it's full, you know, 600,000 plus paid subscribers. Hey, what do you think we should call this podcast? Instead? They asked the AI panel and the AI panel gave feedback of, like, call it. They. I think they gave it, like four options, one of them being the business, not the most imaginative option.

[Kamiko McCoy]

Give me the business podcast. No, thank you.

[Tim Peterson]

That was the. That was the response. And Tracy says that podcast is doing well. So maybe there's something to it that's.

[Kamiko McCoy]

Something to see the overlap between marketing and. And publishers and how they interact. But that's an interesting conversation that I will say for you and Tracy at the end of this. In the meantime, we've got our juicy scoops, which is YouTube versus Disney. I'm going to leave it at that. No, no further explanation. Netflix is putting in a bid for Warner Brothers Discovery, amongst many others, and then we've got some earnings that happened recently. And we'll take a look at how short form revenue is stacking up in that space. So, Tim, talk to me about what's going on with YouTube. I feel like it's a fight. It's a cage match happening here, and I feel like YouTube's winning.

[Tim Peterson]

I mean, yeah, this is YouTube flexing its muscle. Flexing its muscle on the pay TV side of things, more than I think we've seen it do ever since it got into the pay TV business years ago. So YouTube's already become and for years now has been the most watched streaming service on TV screens. And so every month when Nielsen puts out, it's the gauge report, YouTube's at the top in terms of the streaming services. People are spending the most time watching YouTube TV, which is its pay TV service. So its version of a Comcast, a charter, or a DirecTV has also become, like, a really powerful part of its business. And even Moffitt Nathanson, the analyst, have projected that YouTube TV will become the top pay TV provider in the US in 2026. So it's taken on, like, a really powerful role in the overall TV ecosystem, and it's flexing that. It flexed it with NBC Universal and Televisa Univision a couple months ago where it needed to renegotiate the distribution deals with these companies. So, like, as part of NBCU getting its channels on YouTube TV, YouTube agrees to pay NBCU a certain amount of money per subscriber. So, like, if you sign up for YouTube TV and you're paying, I think it's like 70, 80 bucks a month. Some of those dollars, let's say, let's call it $5 goes to NBCU because you get those channels.

[Kamiko McCoy]

Got it. Got it.

[Tim Peterson]

There's always these kinds of. They're called carriage disputes because generally what happens is, you know, historically a Comcast or a Charter would be like, hey, NBC or hey, Disney. Like, we don't want to pay you as much to carry your networks. And nbcu or especially at Disney will be like, okay, but you better pay us to carry espn because people are paying for your service so they can be watching espn. And so there have been, like, times where there have been blackouts. Especially smaller TV network owners generally have their channels stripped from these services because a Comcast, a charter can decide, I don't really need to carry you anymore. Like, that's not really what people are paying for. Maybe I'll lose some subscribers. But ultimately, when this works out in the wash, I think it's going to be better for my business to not pay you that money. Perfect. Subscriber to the point where YouTube TV had this dispute with NBC Universal. They got it resolved just in time for the fall broadcast primetime season and NBCU having NBA again, Televisa, Univision. That has not been resolved with YouTube TV. TU networks are still missing on YouTube TV.

[Kamiko McCoy]

So Disney is happening on top of this.

[Tim Peterson]

So now we got Disney, which, like I mentioned, ESPN is the most important TV network on a pay TV service.

[Kamiko McCoy]

Yeah, of course.

[Tim Peterson]

So this is like Goliath against Goliath. Disney generally gets its way when it comes to these things or finds a way because it got into a dispute with Charter over a year ago because Charter wanted to like be able to have Disney plus be made available to its pay TV subscribers. And eventually they came to a resolution and they got that sorted.

[Kamiko McCoy]

So what's the beef between YouTube TV, what are they, what are they asking for?

[Tim Peterson]

Yeah, so it seems. So this is according to the Wall Street Journal. It seems to be just the length of deals. So usually these kind of distribution deals last for three to five years. And so it kind of gives the TV networks, in this case Disney some assurance of like, cool, I'm gonna be getting this money for the next three to five years. I can kind of count that minus whatever pay TV subscribers will cut the cord in that time. According to the Wall Street Journal, YouTube TV is asking for deals that are only one or two years long. So much shorter timeframe with these deals.

[Kamiko McCoy]

And I would imagine that now we're talking about cutting into how much money I get to count Disney gets to count on in its pocket.

[Tim Peterson]

Well, especially if, you know, so like, not as secure of a deal for Disney. Potentially more leverage for YouTube if in a year or two. So Disney just launched ESPN Unlimited, right, A couple months ago. Conceivably there are people who are just like, cool, I can cancel my cable subscription or my YouTube TV subscription because I'm just going to get ESPN through ESPN Unlimited. Yeah, YouTube's. If. If that does pan out, YouTube would want to be in a position a year from now to be able to be like, people aren't really paying for espn so we're not going to pay you as much. Disney to carry ESPN. If we were paying you $6 per subscriber before, it's actually not worth that because we're seeing that people are just jumping ship for ESPN Limited. We'll give you $3. So YouTube, YouTube TV it would seem, would want to wait for that shoe to drop before it locks into a, like a longer term distribution deal with Disney. Or it may Just be like, this is how things are going with the traditional TV business because you have the NFL which only a couple of years ago renewed its rights deals with Disney, with Paramount, with Fox and added Amazon to the mix. And those are I believe are like 10 year long deals. So they're not coming. Supposed to come up for new for a minute. Yeah, I forget if it was, I forget which publication, but a publication reported like a month or so ago that the NFL is actually thinking about like opening up its renewals all over again. It's kind of like wanting to renew your lease. Now why would you want to renew your lease if you're the tenant because you think you can pay less for rent? If you're the landlord, it's because you're worried that you're going to have to charge less for rent in the future. And so that seems similar to what we're seeing play out here where Disney wants to protect its bag because the traditional TV business isn't going so well. Whereas you, YouTube seems to see an opportunity to protect its own bag or at least protect itself against if ESPN is less valuable to have it its pay TV bundle. And the interesting thing is like YouTube TV is part of. YouTube is part of Google. So it's a tech giant who, unlike a charter or Comcast where up until recently the overwhelming. The bulk of their business was from their pay TV services. And so they were like very dependent on a Disney, especially to have ESPN to have those subscribers. YouTube TV isn't necessarily a side project, like it's an important part of their business, but it's a very small, relatively speaking part of Alphabet's business.

[Kamiko McCoy]

I was gonna ask kind of, you know, who do you predict being the winner and loser here? But again, it's Goliath versus Goliath.

[Tim Peterson]

My sister who has YouTube TV TV and was complaining over the weekend.

[Kamiko McCoy]

I was gonna say the losers are people whose shows have gone dark.

[Tim Peterson]

Yeah, because she was just, she wanted, she's gotten really into running. She just did her first half marathon a little over a week ago. Congratulations to her. She wanted to watch the New York City Marathon yesterday. Was not able to because of this block. I didn't even realize ABC carried the New York City Marathon feed. That's how out of touch I am. But yeah, ultimately like subscribers are kind of always the losers in this case because you're still paying for YouTube TV. YouTube has said it's going to, I think discounted Service by like $20 to make up for, you know, this blackout period.

[Kamiko McCoy]

Yeah.

[Tim Peterson]

But even still like now she's having to decide, does she sign up for ESPN Unlimited? What, like, what move does she make?

[Kamiko McCoy]

Yeah, I mean, I don't know, maybe she can spend all her time running as opposed to watching tv, just swapping one hobby out for another. But it's continuing down the traditional media thread here. Netflix is putting in its bed for Warner Brothers. So talking about protecting the bag. Is WBD gonna protect its bag here? Because they've had a couple of entrants in this space. No, they've had Paramount. I think Amazon, Apple have also put their hat in the ring.

[Tim Peterson]

Yeah. So to be clear, Netflix hasn't submitted a bid for Warner Brothers Discovery, as far as we know. Well, no. So Reuters reported that Netflix is looking into potentially placing a bed for Warner Bros. Discovery. It's, you know, hired lawyers and, you know, financiers, like all the fancy people you would need if you're going to be spending this kind of money to buy a company. According to Reuters, Netflix is primarily interested in the Warner Brothers studio and the streaming business. So HBO Max, as it's called now, not so much in the parts of the business that like, honestly, Warner Brothers Discovery isn't so interested in. Because Warner Brothers Discovery announced earlier this year it's going to split off the Warner Brothers studio and the HBO Max business into one company and then the cable TV network, cnn, Food Network, all the Discovery channels, the Turner properties, those are going to be their own company, similar to what they have and they have Comcast did with NVCU and Versant. Name I still can't pronounce correctly because it's a horrible name. Netflix is considering making a move. Amazon and Apple, Bloomberg has reported, are also considering making some sort of move here. It's a company that has absolute made a move multiple times to acquire Warner Brothers Discovery has been Paramount. It's made three offers and Warner Brothers Discovery has turned down all three of those offers.

[Kamiko McCoy]

Is there any indication here that Netflix could be a stronger offer than Paramount?

[Tim Peterson]

I don't know because like, from what I can tell, Netflix doesn't need Warner Brothers Studios or HBO Max as much as a Paramount would need.

[Kamiko McCoy]

Yeah.

[Tim Peterson]

So I would think, and you could say kind of the same thing about Amazon. Amazon's got its own studios. It's had, you know, that business hasn't been going super great for Amazon. Apple, Apple has a lot of money. It's got a ton of money in the bank and it hasn't been doing so well. Like it's an also ran when it comes to the streaming business. So it could definitely benefit from having an HBO Max and having Warner Brothers Studios. But Paramount would be the most desperate also. There's just the question too of desperate. Any acquisition of this nature, you're gonna have to get FTC approval, you're gonna have to get the government to sign off on it. Do wants to do that dance it makes.

[Kamiko McCoy]

Which begs the one what is it going to take for the CEO and president of Warner Brothers Discovery, David Zaslav, to decide that he wants to make the sale? I would imagine that the strongest competitor here, which he has turned down three times at this point, would be the elephants who are in the good graces of this current presidency. I don't know what this looks like for, for others. So I guess the question here for me is just like, what's it going to take to turn that no to.

[Tim Peterson]

A yes, a bigger number. Because ultimately that's all it's going to come down to is whoever makes the biggest offer. Because David Zeislav, yes, he's the CEO of Warner Brothers Discovery. Warner Brothers Discovery though is a publicly traded company. They answer to their shareholders. If the shareholders are just like, oh, I like that number, we're taking that number. It's really more about this kind of self imposed deadline of the Warner Brothers Discovery split that's supposed to happen next year because Paramount, it's been reported, is cool with buying the entire company. According to this Reuters report, Netflix is only interested in one part of the company. And so then it becomes a question for the shareholders, which are they gonna prefer? Selling the whole thing and getting their money or getting less money for just part of the business and then having to hold on to their shares in the cable TV network business. And that sounds terrible. I don't know what hope there is for them of that business increasing in value in the coming years.

[Kamiko McCoy]

No, I think we've, we've solved the riddle. We've solved the riddle here, the two of us. I, if I were a shareholder that would be, well, you know what, I'm not a shareholder. They can do as they please. But that seems the most sensical to me. I don't see the traditional media business turning around here anytime soon.

[Tim Peterson]

They think like private equities would swoop in by that business. And so then maybe there is a, like, you know, the parts are greater than the sum.

[Kamiko McCoy]

Perhaps, perhaps one thing that has proven out to be a winning strategy is short form videos. That is something that we can rest our laurels on, know because based on the latest earnings that's been A big get for meta and YouTube and whatnot.

[Host 1]

Yeah, yeah.

[Tim Peterson]

So during Meta's most recent quarterly earnings call, CEO Mark Zuckerberg said that Reels has hit an annual revenue run rate of more than $50 billion. Now one thing we got to note here, annual revenue run rate is one of those very squishy metrics. This is not, it's not really a real number. What that's saying is like, and it's also unclear like what the basis of this is, but it could be something I was like, wow, if we took all the revenue that Reels brought in in the past week and we projected that over a 52 week year, it would equate to $50 billion. I don't know if this is based on a week, I don't know if it's based on a month, I don't know if it's based on like a full quarter. But anytime like annual revenue run rate gets thrown out as a number, that's immediately my head goes like, not a real number. That said, it's still if this holds the fact that it even got to that on like an annualized basis or like this projection is not nothing to sneeze at. It's not. So Meta brings in around $50 billion each quarter in total revenue. So we're talking like a quarter of the total revenue for the company could be coming from Reels a year from now. That's pretty significant.

[Kamiko McCoy]

Yeah. As many people send me reels to my DMs, I wouldn't be surprised. But also, I mean, I, I, I, to me that makes sense because it wasn't too long ago that we were even talking about like where a YouTube short or where a real stood in comparison to Tick Tock. But now I think they've got some better leverage here. One, because the TikTok ban still exists. Right. And two, you know, does it what.

[Tim Peterson]

Actually is going on there?

[Kamiko McCoy]

TikTok Runway. So hypothetically, maybe down the line, you know, it's ever looming, but there's a couple of factors at play that's really given these platforms some leg room, if you will.

[Tim Peterson]

Yeah, yeah. Because then we also had Alphabet earnings. Alphabet, the parent company of Google, parent company of YouTube. CEO of Alphabet, Sundar Bachai announced on its earnings call that YouTube Shorts now generates more revenue per watch hour than long form videos. Which is wild to me because when I think of YouTube, I still think of long form videos. That it's more revenue per watch hour means that for all the time people are spending watching long form videos, it's more valuable when they're spending time watching shorts. And so there's more of an incentive than for YouTube to get people to watch even more shorts.

[Kamiko McCoy]

That's really, I think, I mean we've been having conversations about like the value of short form content and whatnot in all types of different ways. I think even Spotify now has like short form videos on their app. But I think you've seen it now move to being like a revenue, more of a revenue driver than it has in the past. And these latest, excuse me, earning earnings calls kind of speaks to that specifically for these platforms.

[Tim Peterson]

Yeah. What is going to be interesting is like if YouTube Shorts has become that valuable to YouTube already, where YouTube Shorts generates more revenue per watch hour than a long form video, how does that then affect the trajectory of YouTube when it comes to products plans? It's work with creators. Like I imagine it's still going to support long form creators.

[Kamiko McCoy]

Yeah.

[Tim Peterson]

But is it going to be doing even more to encourage short form creators and kind of catering towards short form creators that could like just by virtue of everything is finite, come at some expense to long form creators?

[Kamiko McCoy]

Yeah, that will be a big question because like a lot of times when I talk to creators, kind of one of their end goals is to get to be a long form creator on YouTube. Right. Because that's where the biggest payouts are and that the longevity of your career and whatnot. So if based on this information, YouTube kind of makes a different step, that does kind of then change the trajectory of like what a, what's the word that I'm looking for the lifespan of creator then looks like.

[Tim Peterson]

Yeah. Cause you've been reported on the middle class crunch for creators, right?

[Kamiko McCoy]

Yeah.

[Tim Peterson]

This puts like YouTube long form creators very much in that crunched position because as we were talking about with YouTube and Disney, you've got YouTube like really getting into high end TV and streaming. With YouTube TV and you know it's carry, it has like the NFL Sunday ticket package, it's carrying live sports and then it has like the Mr. Beast, the Michelle Cars, the Dhar Manns who are doing TV quality programming and then it's got shorts creators who have based on this stat proven to be really lucrative for YouTube. And who in the, who is in the middle? The long form content creators.

[Kamiko McCoy]

Yeah. And if again you've got the, the faceless TikTok band happening in the background, then that takes away a place for you to be creating some content and then you've got you know, meta pushing reels and things. Like that, it really. The walls start to close in on the places where you can show up and not just show up and be top of mind for your audiences, but also to make money, which I think is kind of the. The big get here.

[Tim Peterson]

Although, I mean, what's that saying? When one door closes, a window opens. Sora.

[Kamiko McCoy]

Yeah. Start plugging away right. Right now. Which takes us back to how the circumventing AI, AI has kind of presented itself for publishers, creators alike as a brick wall. You can go around it, over it, through it, under it, doesn't matter. But back to your conversation with our guest this week, Tracy from the Times and Sunday Times. She talked a little bit about her own ways around AI, working with it and through it.

[Tim Peterson]

Yep, exactly. AI is a threat if it's a very real threat. But there's a lot of ways in which AI is a tool. What the Times is doing with what they call synthetic research panels, what everyone else that I've talked to called synthetic audiences, is definitely using AI as a tool and in a really interesting way. So the folks at the DJ Publishing Summit Europe really enjoyed this conversation. I hope the listeners to this podcast do as well. Oh, and before we go. So we will be on hiatus next week, the week of November 10th, so no episode on November 11th when we would normally release an episode until. We will be back with you with the November 18 episode. Thanks, everyone, for listening.

[Host 1]

Hey, Tracy, thanks for joining us.

[Tracy Averbon]

Thank you for having me.

[Tim Peterson]

So I think if, if Digiday were.

[Host 1]

A tabloid, the headline on this session would probably be something along the lines of the Times is using fake audiences for its marketing not quite the case at the same time, kind of the case. So we had on the Digiday podcast a couple months ago, Seb was on, as well as our senior media buying editor, Michael Berge.

[Tim Peterson]

And one of the things that Berge.

[Host 1]

Mentioned is he was talking to media agencies in the US who are using synthetic audiences, which was a new term to me at the time. But basically they were using AI to take existing human audience databases, training the AI in that to create AI versions that they could then query as focus groups. You all are doing something similar at the Times? We are.

[Tracy Averbon]

We are, yeah. So, yeah, I get very nervous when you use the word synthetic audiences we think about as synthetic research and a way to get to market quicker, get an answer quicker, take the guesswork out of. Out of kind of what we're planning to do or thinking of doing. And we've been working on this for a while with several different partners how long? So it's been about, I'd say, eight, nine months in the work.

[Host 1]

This is the long time in the AI.

[Kamiko McCoy]

Yeah.

[Tracy Averbon]

So it's not something you just switch on. So it's a bit of background context. So the Times and the Sunday Times have a huge amount of rich data on their existing audience and subscriber base. Like the ft, we have typically had a, for many years, a hard paywall. Actually, we were the first publisher to go behind the paywall 15 years ago. I feel like everybody that works for the Times says this at some point at a conference, but as a result, we've got this very engaged audience. We know a huge amount of it.

[Host 1]

How many subscribers do you have at this point?

[Tracy Averbon]

So paying subscribers? Subscribers. We have 642,000 subscribers. We don't count people in trial. So in your first three months, we don't count your subscriber. We also don't count secondary accounts, so the bonus accounts. And so there's far more.

[Kamiko McCoy]

But we just.

[Tracy Averbon]

So when we're talking about subscribers from that core base that are subscribed to us and paying. And so as a result, we know a lot about them and we know how to serve them really well, typically we've known less about our guest audiences and our prospects. So what we wanted to do is try and understand their motivations. We're not as advanced as the FT yet. We don't have an AI paywall yet. We have a progressive paywall. And we're working on that. We're very inspired by what they're doing. It's really encouraging, actually. But what we're doing is we are using our own reader panel, and then we are using synthetic clusters and audiences to run these tests side by side to see how accurate it is. We're actually working. We've tried a couple different partners. We've actually working. The one that we have found that works, we believe in the efficacy of what they're doing, is a company called Electric Twin.

[Host 1]

What was it that made them stand out to you?

[Tracy Averbon]

I think it was their approach. So a lot of these synthetic research models, they use data and LLMs. And the difference between Electric Twin and them for us was that layer of behavioral data. They've also added in. Sorry, I get this right. I wrote this down. So the data they use is basically market research, publicly available customer data, government data that's available to everyone. Then there's the models that sit behind that. So the models behind ChatGPT, all of the models, but then what they do is they overlay all of the social science behavioral data on top of that as well. And that goes into the model.

[Host 1]

And so what they, what are examples of the behavioral data? Just to help me understand.

[Tracy Averbon]

So leading academic research, any of those behavioral data that goes into all of the leading kind of academic research to help people behave. Okay, so you know, all of that scientific, scientific research.

[Host 1]

So like interests in terms of like content categories, maybe even shopping behavior, things.

[Tracy Averbon]

Yeah, like anything that's widely available. And so what they've done is they've made a clone of our audience panel or subscriber panel that we can test the accuracy against.

[Host 1]

They've also subscriber panel a subset of that 600.

[Tracy Averbon]

So that's what we've got our own data that we poll our own subscribers and then they made a clone of that so we ask them questions and really together and so we can check how accurate it is. And then they've also made a clone of the news reading population of the UK for us. And then they've looked at that over time as well. So this is, these models take a long time to make sure that they're accurate, make sure that they're right. We, you know, we've had to do like going to our reader panel can take, you know, a couple of days, a couple of weeks to get the data back as well. So just training those models side by side. So and when we looked at how like the first thing you think about synthetic researchers is, you know, can I trust it? You know, how accurate is it? Pretty accurate as it turns out. So let me just, I wrote this down because my head of audience insights will kill me if I get this wrong. So when evaluating the accuracy of the data set, what we did was we used a 10% holdout group on our own audience that we knew that we could trust. And it's about, you know, I think there was a, a score of 0.918, which is about 92% accurate. Now you know, general research is 90 is only 93% accurate.

[Kamiko McCoy]

Right.

[Tracy Averbon]

So for example, if you ask someone a question in a survey at the beginning of the survey, it's a long survey, you ask them the same question, at the end, they will probably give you a different answer. Right. So general research is only 93% accurate. This is 92% accurate. So you know, we are not, you know, we are using this to kind of help us, you know, get, get to get to answers quickly, questions quickly. And a good example of that is really recently we were launching a new business podcast. So we asked our reader Panel and we asked our synthetic research panel, which was in a news reading population of the UK who listen to podcasts but are not subscribers at the time.

[Host 1]

Okay. Which is important for an acquisition.

[Tracy Averbon]

Exactly, exactly. Because it's slightly different audience and yeah, they helped us come up with the name.

[Host 1]

What was the name?

[Tracy Averbon]

The name is called the business. But yeah, sounds pretty obvious. But we gave it about four different names and that was the one came up pretty high with our reader panel, but it came really, really high with the demographic that we were really trying to target for this particular podcast. So that's kind of like a great example of.

[Host 1]

Has that podcast launched yet?

[Tracy Averbon]

It has, it has.

[Host 1]

How did it do in terms of like downloads and subscribers doing really, really well. I did it down to the average.

[Tracy Averbon]

Yes, it's doing really well actually. It's probably our biggest. I mean it helped that the, the, the person they interviewed on the, on the, on the first episode was Sir Jim Ratcliffe. So who barely didn't get that many interviews or podcast interviews. But that was great because it's really, really great. I think it really resonated and helped us shape our marketing. What did the, what was the creative going to look like? You know, you know, most podcasts are video now as well. What did the set look like?

[Host 1]

Did you query the synthetic research on that as well?

[Tracy Averbon]

Not on the set. Yeah. So we haven't got to the point where we are putting images in there yet, but it helps us kind of help build out the narrative. Like, you know, we fed in kind of who was the target audience, what we were trying to achieve from that and it gives a very, very simple answer. And resonating looks really great in the marketing. I don't have the actual number of the downloads as well, but tune in.

[Kamiko McCoy]

It's very good.

[Tracy Averbon]

So, so, so that's kind of like a light touch way of just validating like we had a couple of names that we, we were happy with, but what was the audience going to resonate with as well? And another way I think that it's really interesting for us is really like asking, like helping us think about products differently. Right. Asking the audience. I think one of the questions we asked the reader panel was if you could have one new way to explain experience the times that would, that would make you more engaged. What would that be? What would drive deeper engagement? And again, the difference in our reader panel research and what the news reading population of the UK that would look like, our prospects are were very, very different.

[Tim Peterson]

Really?

[Tracy Averbon]

Yeah. And so our loyal readers are very Skeptical about AI, they're very respectful of personalization. So a lot of the questions we're asking around deeper personalization because obviously that's a big theme and threat and trend. And I think as publishers, particularly kind of, you know, quality journalism as well, you want to be really careful, right? About, you know, the, the efficacy, the, you know, the trust that you've built in your brand. And you know, one of the concerns from the editors there are, you know, people pay for the edit, right? They pay for the human curation. Do they really want summaries? Do they really want personalization? Do you know, what do they want from us when it comes to AI? Certainly not in how we write or how we fact checked or our journalism. But what do they really want? And while our subscribers are a little bit skeptical, they worry about algorithms optimizing towards engagement. They're missing something. The prospect audiences were very different, so they wanted different types of features, they wanted summaries, they wanted like AI explainers, they want more video. And so it's about, so it's about understanding how do we continue to serve our loyal readers where we really know what they want. But also while we're bringing these new audiences, you know, how do we build products and services for them and keep.

[Host 1]

Them engaged in the same way without siloing the audiences. But it's completely different experiences.

[Tracy Averbon]

Exactly, exactly. So what is the level of personalization that's not jarring, that's helpful, it's useful for utility, but in the same way maintains integrity. And we were talking this morning about that article that came out yesterday about, you know, the findings from the BBC and the ebu.

[Host 1]

Yeah, like a European business.

[Tracy Averbon]

Yes, yeah, yeah, exactly. You know how, you know, not only are, you know, a lot of these, you know, LLM, you know, stealing publisher traffic, but you know, the publisher also being blamed when they get it wrong? They're getting it wrong a lot.

[Host 1]

So yeah, I think it was like.

[Tim Peterson]

What more than a third of the.

[Host 1]

People who were surveyed as part of that said the publisher should be partially.

[Tracy Averbon]

Responsible for the hallucination of these, of these, of these models. So that obviously it's a concern amongst editors, it's loyal subscribers where newer. I know, it's like death by a thousand cuts. What else?

[Host 1]

Media.

[Tracy Averbon]

Yeah, so it's always watching that, making sure that the integrity of what we do is always maintained. And then interestingly enough, the synthetic audience research came back with, well, they worry about editorial bias, you know, so maybe the question is in, you know, AI versus human, right? A bias is it is.

[Kamiko McCoy]

It.

[Tracy Averbon]

Are we all programmed to optimize towards engagement? Okay, so that. That poses a very Interesting, interesting.

[Host 1]

So you have your loyal audience who's skeptical of A.I. you have the news reading audience who. The prospect audience who's skeptical of humans.

[Tracy Averbon]

Yeah. Who are typically, typically on, you know, we all obviously want to get those, you know, younger millennials subscribing to quality news, you know, and, you know, Gen Z, you're a long way off.

[Kamiko McCoy]

Yeah.

[Tracy Averbon]

But, you know, but just really, really building those foundations and look at where they trust and look at what they look at. What they want to interact with is more video, it's more voice, it's. They want summaries. They want, you know, instant gratification to get to that answer quickly. How do we, you know, what the synthetic research is really good. It's like, how do we understand them better and then figure out how we infuse some of that into our product building in a way that stays true to the integrity of what we do and not alienate our existing loyal subscriber base, but also bring these people in in the right way.

[Host 1]

Have you implemented AI summaries on articles?

[Tracy Averbon]

No.

[Host 1]

Okay.

[Tracy Averbon]

We have not yet.

[Host 1]

Are you thinking about.

[Kamiko McCoy]

We are, we are.

[Tracy Averbon]

We're looking at sort of, I think the areas where. And again, synthetic research helps us get this, where we think it could be valuable is that deep context. So that's something that our subscribers really want. So they'll go down a rabbit hole on a topic like, you know, Ukraine, for example. You know, they'll go, you know, wanting to know the history of the Donbass. They really want to know more. So how can you help them go deeper through. You know, we have an archive of 240 years at the Times. Not all of it's digitized, but a lot of it is. Like, how can you go right back to understand that what is the video, what are the podcasts, what's all the other material? If someone really wants to go deep on something, they can do that. So we're looking at building out features like that for subscribers. So I think that would be great. And also that that's deep interest to, you know, a lot of new subscribers, a lot of educational institutions, you know, on. On a kind of. The more kind of larger corporate side of the business as well, are very interested in. In going deep on context. But I think, yes, I think people, there's so, you know, we produce so much journalism. I think people do want to know if I miss something. It's often complained to subscribers like, you know, I can't believe I missed this article, you know, so it's like, how do we infuse it in a way that's, that's helpful.

[Host 1]

Yeah, yeah. Because I think it was at our US Publishing summit in Miami in September where I believe it was when we had the CEO of Bloomberg on stage, she was talking about how they had implemented AI summaries on articles. And I forget the exact stat, if you sign up for a Digita subscription, you can watch the full session on the Digiday website. But she talked about how after they implemented AI summaries, they saw that actually lead to more dwell time on pages, also more pages consumed per session. So there definitely is a correlation there with the synthetic research panel. So you modeled that based off your reader research panel, the human panel. Was that a one to one? How did you curate what information to provide to Electrotwin to then create that synthetic research?

[Tracy Averbon]

Yeah, so they were embedded in our teams for quite a few months. And I think this is the thing with a lot of this AI, it's something you switch on. Right. So a lot of it depends on the quality of the information and the data that it gets. So their team set with our data teams, our insights teams, they looked at the panel, they looked at all of our data, they wanted to make sure that they could replicate the model in the best way possible. And then we trained side by side. So we would ask a question, they would ask a question. Then you know, so there's a lot of work went into. So we feel like we've a good degree of trust in that. And then the news reading population uk there's so much data out there, publicly available data, plus the behavioral data as well. To run that alongside along that, run those alongside each other. So, so we're kind of at a good place. But again these are, you know, there are other providers out there. If anyone has a WPP client, they have a tool called Open and they have loads of synthetic research tools there. They don't go as deep as this, but it will get you a quick answer, it will get you. So definitely something you should play around with. And a lot of these vendors, you know, if you're serious, they will give you a three month POC that you can try. Like at least, you know, you can go proof of concept. Yeah, yeah. So it's definitely worth, worth trying. You know, how much does that cost, the proof of concept? Normally they normally do it for free. For free? Yeah.

[Host 1]

But for like a full sign up like with Electro Swin Is it a.

[Tracy Averbon]

Totally depends because it's in SaaS model. Number of seats is what they.

[Host 1]

How much are you guys paying?

[Tracy Averbon]

I don't actually know. I couldn't tell you because I don't know how many seats we have because it's across all of News uk. So it's not just us at the Times using it, our colleagues across the Sun Broadcasting, Wall Street Journal that have access to it as well.

[Host 1]

So was there a like cohort of you within the organization that had to do like the selling to the C.

[Tracy Averbon]

Suite to get the Times? Yeah, the Times kind of were the first to adopt it and now we've. We were in a long POC with them, but I think they, you know, and then News Corp. News uk, I think they were an investor in the circumstances seed round for the, the, the vehicle that are investing behind Electric Twin as well.

[Host 1]

Okay, so that's helpful.

[Tracy Averbon]

So yeah, they were quite keen to do PSE with us, which is, which is good. But we definitely believed in the, in the, in the efficacy of, of, of how their approach to it.

[Host 1]

Was there anything from your side that like kind of cinched the deal and selling it to leadership to invest in this?

[Tracy Averbon]

Just real time showing them the model in real life, you know, just sitting there like they've got on your phone asking it any question, you know, questions on like price elasticity. Fascinating. You know, doesn't always get it right, but it's like it's directionally or things that you're thinking about doing. Yeah.

[Tim Peterson]

So who all in the organization has.

[Host 1]

Access to be able to query the panel? Because I feel like I would just spend all day just asking questions.

[Tracy Averbon]

Yes. So you've got to be. So this is why it's good as a SaaS model. So our data and Insights team have, have the access to it. I think some marketing has the access to it as well to a lot of our marketing research as well. So that's generally who we would say. Because otherwise, you know.

[Host 1]

Yeah, because are you getting billed on a query basis?

[Tracy Averbon]

Weaponize it to tell you anything? Oh, sure, yeah yeah.

[Host 1]

Are you all being built on a per query basis?

[Tracy Averbon]

No, I don't think so. I think it's just a.

[Host 1]

Are there any like quotas or limits in terms of like on the different teams of how many queries they can be submitting in a given day or week?

[Tracy Averbon]

Not on, not how we're working. But I'm sure there's lots of different ways you can like per query or SEATS or different SaaS models that they operate. But I think we've got, I guess an annual contract where we've got a set number of seats and that's how we interact with it. But yeah, you can have a debate with it, you can do basic polling. So yeah, it's a helpful tool, you know.

[Host 1]

And how do you kind of keep it as a helpful tool as opposed to like overtaking? I can imagine there being meetings in which everyone just like, well, I was talking to the Synthetic Research Panel and this is what they said.

[Tim Peterson]

Well, when I was talking to them.

[Host 1]

They actually said this instead. And then you're kind of losing the humans in the loop model.

[Tracy Averbon]

That's why we leave the data and insights team to, you know, so you've got, they get a lot of requests. Can you just ask this thing or can you. But that's why we give it to them to, you know, because again, you know, people will always find a way for data to weaponize data to make a point or an argument. It's no different to looking at different research studies. And, you know, so but I think you've got to recognize that, you know, you've got to use alongside your own common sense, you know. And again, like any, like, you know, approached AI two years ago, like we, you know, the questions we were getting asked from, you know, our CEOs was just like, what is your AI strategy? What are you doing? And now we're at the point where like, you know, we're really focused on our business strategy and then how we can use AI tools to supercharge that and getting out of the opposite as well. So it's not used in every instance. It's used when we like, we think it will be useful to help us get to a decision. It's really great for stuff like campaign creative again or naming, just getting us to those points quicker.

[Host 1]

Okay, we're over time. I'm going to take a little bit of time for this next question because I'm curious. So as much as you're able to train this on the existing audiences, existing subscriber base to get prospects, I also am curious if you're able to train it on maybe lapsed subscribers or people who have turned out to identify churn patterns and then be able to query based on what's the effect going to be on lifetime value or how do I identify if someone's potentially going to churn soon?

[Tracy Averbon]

Yeah, we can. We're not doing that at scale yet, but that's definitely something that we want to do. So taking these cohorts, understanding them a lot better because we do machine learning for that for some sort of predictive churn modeling anyway. But it's getting us closer to understand just not like why someone churned but when they're going to churn. So that's really helpful as well. So we can actually put the appropriate interventions in there. So whether it's what content to serve them or do we need to get an offer in front of them or do we you know, but then is implementing those interventions as well that's to get on a product roadmap. So these things take time. We're in the kind of process of doing all of that now. You know, where else where it's also we're not really so much using synthetic audience research to help us here, but in our performance marketing we're using we actually want a Digiday Technology effectiveness award for our use of like creative automation as well. So we will like we will, you know, we may have used synthetic research to kind of like pick a piece of campaign creative or a set of creatives. And then what we use is a tool that helps us quickly create multiple sets of creatives that when we can use in performance marketing and can really bring down your cost per acquisition for a customer as well. And all of that goes back in to the models just to try and help that make it more efficient.

[Host 1]

It's really fascinating stuff. We're unfortunately over time but I could have spent another hour at least on stage and I'm glad it's with you as opposed to with the synthetic reason surgery. Tracy, thanks so much.

[Tracy Averbon]

Next time we'll send thank you.

[Host 1]

Thank you.

[Tim Peterson]

Thanks for listening to this episode of the Digiday Podcast. If you enjoyed it, please leave us a rating and a review on Apple Podcasts, Spotify or wherever you're listening. Get more from Digiday with our daily newsletter sent out each weekday morning. Visit digiday.comnewsletters to sign up.