← The Digiday Podcast
Loading summary
Transcript68 lines
- [00:00]
Kamika McCoy
Foreign.
- [00:10]
Hello. Hello and welcome to another episode of the Digiday Podcast, a show for anyone who's wondering if ChatGPT is going to be how we get all of our news in the Future. I'm Kamika McCoy, senior marketing reporter here at Digiday.
- [00:22]
Tim Peterson
And I'm Tim Peterson, Executive Editor of Video and Audio Digitay Media. Kamika, do you think Digiday.com should have an AI chatbot?
- [00:32]
Kamika McCoy
You know what? I have, I have never met a news publisher's search functionality that I liked, including our own. So perhaps. What about you?
- [00:41]
Tim Peterson
Yeah, I kind of land that way too. But what was interesting is, so last week I was at the DJ Publishing Summit in Vail, Colorado. We had our other publishing summit in September of last year and a big thing a lot of the publishers talked about in September was AI generated summit memories and AI chatbots being used as new and improved on site search. This time around though, at last week's cps, the conversation was more around using AI as page editors on site. So for this week's episode we had Chris Moran, who's the head of editorial innovation at the Guardian, and he talked about why the Guardian decided not to to create an AI chatbot as its first reader facing AI tool.
- [01:31]
Kamika McCoy
Interesting, Interesting. That sounds like a very juicy conversation for everybody in the audience and for our listeners as well.
- [01:36]
Tim Peterson
Yeah, yeah, I thought it was like a really interesting. Because he gets into the principles the Guardian wrote up to guide its AI product roadmap or just its plans. And then we got like really into it with this new product, the first reader facing AI product that the Guardian rolled out. We even did a reveal of it on the stage, which is gonna not necessarily translate to audio only form in the show, but basically if you look on the Guardian site for this feature, it's called Storylines, it's not super obvious that it's an AI feature. It's not like an AI chatbot pop up or anything like that. It looks like it's just a related content module. It is fully disclosed on there that it's AI generated, but it looks kind of like the rest of the page, which is what makes it interesting to me.
- [02:39]
Kamika McCoy
Yeah, going back to my original comment, I think news publishers have for a long time struggled with search functionalities on their site. So to see some of these publishers start to tackle it after many a moon, no less, very interesting. And I'm excited to see how they get into it.
- [02:54]
Yeah, yeah.
- [02:55]
Tim Peterson
So Chris and I, we really get into it in this conversation and he like makes the point around how one thing large language models are good at is pulling narrative threads from data, which is effectively, that's what articles are. And that's why using an LLM to create this effectively like related articles module makes a lot of sense. But he does a much better job than I can of articulating what the Guardian built here and why they built it and how they tested today. So we get into all of that in the conversation.
- [03:30]
Kamika McCoy
Fantastic. Well, with no further ado.
- [03:39]
All right, so let's talk about AI products. So I feel like every publisher, if they haven't already, is trying to figure out, do we start throwing AI into the product at some point, into the reader facing product, the consumer facing product? I'm sure many publishers are already throwing it into like the back end products at the Guardian. How have you. When did you all start thinking about this?
- [04:04]
Chris Moran
So, like most news organizations, it was probably a couple of weeks after ChatGPT first came out. And our particular approach, which I'm really proud of and really pleased about, was we went principals first, first of all, and I think that matters to us more than anything else. We wanted to create a kind of frame in which we considered this.
- [04:25]
Kamika McCoy
What are the principles?
- [04:26]
Chris Moran
There's just the three, they're quite simple. One is that anything we do should be for the benefit of our readers. The other should be that it's for the benefit of our staff and our mission. And the final one is with respect to copyright and content creators. And those three very simple things were designed to kind of stand the test of time. And they have, and they've genuinely kind of helped us build journalistic values into the system. Because, like, you know, we're all always talking about AI, you know, and if you go on X or Blue sky, most of the conversation is about what you might miss out if you don't move now. And I very strongly remember the first person I wanted to speak to when I was thinking about principles was a guy called John Norton. He's a professor at Cambridge and an observer columnist. And his advice still rings in my ears. He said, take a deep breath. And I think that still guides everything we do. We're not going to die if we don't build a chatbot tomorrow. We need to be really clear about what the threats are externally. But ultimately what we have is something that's worth protecting.
- [05:33]
Kamika McCoy
And the principle, like the first two principles, the first principle definitely that it'd be to the benefit of readers. That feels like anytime I talk to a brand marketer and they're saying, well, we've made this pivot to a customer first strategy. It's like, well, what the hell was the strategy before? That makes sense to the benefit of staff. As someone who's staff at a publisher, that's just reassuring to hear the copyright one feels trickier to actually hold to that principle though.
- [06:02]
Chris Moran
Yeah, I mean, I think if you're thinking about this, this was kind of. We were writing those principles three months after ChatGPT came out. So the conversation at that point was very much about how is this going to impact us? And that led into like with everyone, a whole stream within the Guardian that's focused around licensing and what licensing now means in this world. Right. So a huge part of it was initially self preservation, but also we want to take it seriously in terms of us treading on other people's work. Right now the argument of course is if you use an LLM, then training and everything else. But those arguments going to have to play out like with the New York times lawsuit against OpenAI, I think. So a big part of our focus obviously was not training, but grounding.
- [06:52]
Kamika McCoy
What's the difference?
- [06:53]
Chris Moran
So training is what goes on before the model is released and then at the point where it cuts off, it still needs access to new content. And that clearly felt a space in which we could operate. And that was one of the things that led to us having to deal with OpenAI as well.
- [07:07]
Kamika McCoy
Okay, okay. So started late 2022 into 2023, you're all establishing these principles. At what point do you start to get some sense of what is the first reader facing AI product you want to bring?
- [07:21]
Chris Moran
So initially, just to be clear, it was very small scale. It was me and one developer that's now scaled up to a team that's always at least two, and sometimes four, along with data science. But our remit again was something we argued for very strongly, was that we should have the ability to not deploy when we chose not to. So the first thing we built was a chatbot.
- [07:43]
Kamika McCoy
Sure, why not?
- [07:45]
Chris Moran
Internally. And the main reason we did that was not because we were completely convinced that we should release a reader facing chatbot. It's because we wanted to see what that looked like and what it would output and how we felt about that.
- [07:59]
Kamika McCoy
Did you have any flashbacks to the Facebook messenger bot days?
- [08:02]
Chris Moran
Oh my God, so many. I mean, also what hangs across the top of all of this at some points as well, I think, is the pivot to video. Right. There are lots of similarities with that conversation. Text is dead. Everyone will now only watch video. And what I hope and it certainly feels like is that our industry has got a better grasp on that generative AI is undoubtedly going to disrupt everything. But at the same time, the idea that what the Guardian offers right now is valueless in the face of that, I think is wrong. It doesn't mean we should be complacent.
- [08:37]
Kamika McCoy
And so this first AI chatbot, it felt like everyone's first AI chatbots was just better on site search. Is that what you all did?
- [08:47]
Chris Moran
A bit. I mean, it did create a summary and that was ultimately why we decided not to deploy it. I mean, I think ultimately the big question, and I'm not disparaging anybody who has deployed a chatbot in a reader facing way, we are all finding our way right now. Right. But ultimately, beyond the editorial legal nightmares of something you absolutely don't control, because
- [09:08]
Kamika McCoy
if it gets the summary wrong, that's a problem.
- [09:10]
Chris Moran
Yeah. And you know, I don't. Just because it's pointing at Guardian journalism doesn't mean it's going to be accurate. What the LLM then spits out. But ultimately, I guess it's a philosophical point, isn't it? If you do point chatgpt or another LLM at a Guardian archive is what it spits out. Guardian journalism, ultimately, I don't think it is, and that doesn't mean that we're not interested in that area. But it does mean right now that it feels like a move away from a set of values around accountability for what we produce. In particular that I think we're not particularly comfortable with.
- [09:46]
Kamika McCoy
Can you dig in on that for a second? So AI summary of a Guardian article isn't Guardian journalism, that's what you're saying?
- [09:55]
Chris Moran
Yeah. Or a kind of aggregated summary of five Guardian articles. Right. Which is often how, you know, which is how our chatbot works. Principally, everyone talks about liquid content and everyone is very excited about liquid content, and we should be. But the thing that interests me is there's not much conversation about what static content is and what benefits it has that are distinct from liquid content. Which might sound strange, but ultimately I think it's quite a good thing that, for example, when we produce an article, everybody who comes to the Guardian sees the same thing. In fact, you could argue that's actually the beginning of a whole other series of things that fall out of that, like community. You can have a conversation about the same experience. You can build affinity with us because what we curate matters. Right. And the choices that we make matter. So if ultimately you're building something which looks a bit like a Google AI overview. That is flexible, it's exciting, but it doesn't express a Guardian point of view. And you are ultimately putting your journalism in the hands of something which you don't control.
- [11:06]
Kamika McCoy
So not an AI chatbot, not an AI overview.
- [11:09]
Chris Moran
Yeah.
- [11:11]
Kamika McCoy
How did you start getting closer on what eventually the product was?
- [11:14]
Chris Moran
So, first of all, like you said, a lot of our focus was actually just on internal tooling. It just felt the safest place, human in the loop. Our editorial code dictates that as well to a certain extent, particularly around the writing and creation of journalism. So we did deploy our AI chatbot internally. It's a really useful resource for Guardian journalists who are trying to kind of work on something they haven't necessarily worked on or pulling strands together.
- [11:43]
Kamika McCoy
Super helpful because you can trust them to then go and actually read the articles.
- [11:48]
Chris Moran
You sincerely hope so, right? I mean, I think one of the most interesting thing about AI principles in general and guidelines in editorial organizations is you could very strongly argue that a journalist's job and their expertise and their experience and their values should actually ensure working with AI is safe. Check your source. Don't just take one source, don't just replicate it. Think about plagiarism, think about how it factors into what you're actually trying to say. So most of our focus for a long time was on internal tools, but it just felt like we needed to move into this area. So we were looking for something that was very controlled, that wasn't going to produce much text that definitely brought value to a reader. And where we landed was something that was quite narrow and focused. What we're going to talk about is not going to save the Guardian from a wave of AI kind of existential angst and risk. But I think the way that we've thought about it offers a kind of avenue for the kind of things, multiple different products that we could build to make the Guardian a better place for a reader.
- [13:00]
Kamika McCoy
When did that change happen, that you decided, okay, we have the internal tools, but now we do need to do something reader facing.
- [13:06]
Chris Moran
So it happened probably about a year and a half ago. We were starting to think about it in earnest, but we had a few big projects to finish off. And where it really came from actually was a workshop with Google where they basically were just challenging us to think about what a reader facing product might look like.
- [13:27]
Kamika McCoy
So they could steal the idea?
- [13:29]
Chris Moran
No, no, it was just a kind of general workshop with them. And in fact, the team that worked on this were funded by Google News initiative. And where I landed was something I've been very interested in for a long time, but almost nobody else is. And that is, we call it a tag page. Most people call it topic pages, and basically most big news organizations have them. Right. It's the archive page, where you just, typically speaking, have a wall of reverse chronological articles.
- [14:01]
Kamika McCoy
Right.
- [14:02]
Chris Moran
And I remember when we first built these back at the Guardian, because I'm ancient at this point, and what I remember about seeing them first was how exciting the promise of these pages was for an organization which, typically in the uk, I don't know, it's the same in the us, but we talk about newspapers as being tomorrow's fish and chip paper. You create content for the day and then it's effectively gone. So what excited me about those pages was this could be a way of leveraging our archive and actually showing it to human beings in a way that was useful. But in fact, what happened was everyone realized it was impossible to edit all of these pages. We had 27,000 of them. So you end up with just this wall of article, article, article, article. So somebody might, I don't know, land on a page about Mexico from the Guardian, and many people do, and they might be expecting to understand something more about Mexico, whereas in fact, what they see is just a load of unstructured content. So those pages have real promise, but broadly speaking, they've never really realized it. And what excited me was this might be the moment where the technology had arrived to actually build on that promise.
- [15:17]
Kamika McCoy
Everyone in the crowd. So it's not a chatbot, it's not AI overviews. Do you all have any sense of what this thing might look like, or is your head just kind of swimming with it? Should we show them?
- [15:28]
Chris Moran
I think we should probably show.
- [15:29]
Kamika McCoy
So bring it up on the screen in a minute here. We're going to scroll down a bit to this storyline section.
- [15:37]
Chris Moran
In fact, I'm going to annoy you. Is it Mark backstage? Do you mind scrolling back up first of all, Mark, Is that right, Mike? Thank you, Mike. This is what a tag page typically looks like on the Trump administration. This is the most recent content and you can see it's just a wall. There's nothing here to help people understand or get a kind of grip on what on earth is going on in this huge topic. So now if we scroll down, this is a very small AB test. This is what we've been working on. We call it storylines. What I think the AI enables us to do in a way that is automated that we couldn't do manually, is to decode this page by using the idea of narratives. So the technology is doing one thing first, which is quite straightforward. It's generating from a list of the most recent 200 articles on this tag what it thinks the three big storylines are right across those articles. And it's really good technology for this. Right. It's good at applying narrative to something and decoding it. But what was important to me was that it wasn't just creating some kind of summary. So the only text that the machine is actually creating is the titles. Those 1, 2, 3, ICE, immigration enforcement and detention, US war with Iran and RFK junior health policy changes and even just that, I think when you land on this page about the Trump administration is just a really helpful bit of context. Even if nobody clicks on anything else, like these are the three big, chunky, meaty stories that are going on under the hood of this art of this topic. Then what the machine does is it identifies the four most useful news pieces and that's what you can see in that big first tab. And then if we can scroll down a little more Mike, then a simple Agentix system then goes off to find great opinion pieces, a relevant deep read. It looks for multimedia profiles and interviews. If it can't find something relevant, then it won't show something. But in this case, you can see, unsurprisingly, on this particular very recent topic, it's actually found quite a lot.
- [17:53]
Kamika McCoy
It's interesting to me. So when we had the publishing Summit in Miami in September, we talked a lot about AI powered products publishers were implementing. Felt like the most common examples we were hearing across those sessions was AI generated summaries. We're two sessions into this publishing summit, but it feels like it's AI powered page editors, really.
- [18:16]
Chris Moran
Yeah. It's a curatorial tool fundamentally. And what it's designed to do is highlight the journalism and showcase the journalism, which is of course what our biggest value is. Right. The summary thing always really fascinated me. Loads of people have done that kind of thing where it's like you land on an article page and there's a little button saying, would you like an AI summary? And again, I'm not disparaging that because I think lots of us immediately went, oh, it's quite good at summaries and what can we do with that? But ultimately, I think that's where the kind of reader use starts to get a bit blurry. You land on an article and what do you see? You get a headline, which is a summary. You get what we call a stand first or a trail, which is a summary and you get a paragraph at the top which is probably a summary of the most important aspects of the thing. So offering a reader an opt in summary on top of all of those three things, I think it's something we never felt like the need to really pursue. I think that there's more to say about this. I think one of the interesting things is how much we are controlling the machine.
- [19:19]
Kamika McCoy
So I was curious about that. What's the supervision here for? For example, this is about ice. Immigration and Customs Enforcement. Is there someone making sure that an article about the ice bucket challenge doesn't pop up here?
- [19:32]
Chris Moran
So I think there's a few things. First of all, the way that we've built the machine is not probably what you might expect. So one of the things that we're trying to do is experiment with newer technologies or specific technologies so that my little team can then pass that onto the wider kind of product and engineering teams. So we were going to look at vectorization for this. You throw everything in the pot, you allow the machine to extract kind of meaning and relationships from basically the idea
- [20:00]
Kamika McCoy
of taking a keyword, assigning it coordinates and multidimensional spaces, and then based on proximity, that's how you determine this. ICE is related to government is not water ice.
- [20:12]
Chris Moran
Exactly that. But one of the things that we realized quite quickly was if you just throw all of that into the pot, you're allowing the machine to make those connections much more than the connections that you have made editorially. So in fact, we are not showing the LLM, the articles, the body copy at all. We're only showing it the headlines and the trails. And that's very, very intentional. It has a kind of logistical, practical aspect, which is if you are only showing 200 headlines and trails, you're also limiting the context. And you can be really confident that it's not going to lose its way or miss something within that. But more importantly, what you're saying to the machine is I want you to pay attention to what we said. This is about what our human editors said. Said this is about. And really particularly for a curatorial tool, that's great. It works really effectively and it stops, I don't know, the machine from noticing a reference to Brad Pitt in the 17th paragraph or something. And assuming it's about that, did you
- [21:10]
Kamika McCoy
all have to do anything to change any headlines? Granted, Guardian, you're a general news publisher, most of your headlines are pretty straightforward. But I don't know if you all had the upworthy era where it's like you won't believe. And so then the LLM sees that and like, oh, well, I can't recommend that article because people won't believe it.
- [21:28]
Chris Moran
No, my background, I started at the Guardian by building the audience team there. So we have a very, very long history of intelligent optimization, digital optimization. We never fell into that trap. And that's partly because we don't just care about page views, we care about the amount of time somebody spends on. On the page at the same time. So pre that, you do have a huge load of content that is not greatly signaled, but typically, because we're looking at 200 articles on a given tag, you're not going to be getting that far back. So in this case, to provide something useful to the reader, you're almost certainly not diving back into 2007.
- [22:13]
Kamika McCoy
What was the process for? Did you have editors training or involved in the training here?
- [22:19]
Chris Moran
Yeah, and I think that's another massive question and a massive challenge for newsrooms who are genuinely interested in building some kind of editorial oversight. If you're going to trust a machine to do this, if. And again, we do have some guardrails there. What you're going to need to do is get an evaluation process where you're actually getting expertise from the newsroom applied to the outcome. So a significant portion of the work that went into this was rooted in an evaluation period, with 20 of our most senior editors evaluating multiple instances of these to capture exactly what was going
- [22:57]
Kamika McCoy
wrong, what did that look like? What was the process for each editor?
- [23:00]
Chris Moran
So our data science team have become very good at building this in a practical and logistical way. But it is a real. It is a real challenge, and I don't think enough of us have probably taken it seriously enough. But in this instance, we were asking them to look at a set of at least 10 of these across the course of two weeks and giving us quite detailed feedback. But that was very, very focused as well. So, first of all, most importantly, are any of the storylines it's identified just badly wrong? And also, what's the text? And is there something which typically we would avoid in that kind of headline? And then what does the linked content look like? Are there red flags in something that looks tasteless or unpleasant or unfortunate in the way that it's kind of next to each other, or is it just irrelevant? And that gave us an enormous amount of rich feedback, which we then went straight back into the prompting and the machine with. And started tweaking things. One very simple thing was, originally we were asking for four storylines and the Most common feedback was that typically one of those storylines was that felt like the machine was reaching for something and that had a whole series of different kind of implications for the content that it was finding. So in that instance, the simplest and best thing to do is just ask it to look for fewer. But we also got a lot of detail back about how long the headlines were and whether or not that was allowing mistakes to creep in or just kind of language that we would not typically encourage. Again, one of the best bulwarks against that was by asking it only to look at headlines. It's ingesting stuff that has gone through multiple layers of editorial checking. So it's not going to typically start using language, which we wouldn't. But we worked on that a lot. As you'd expect. Just to be clear, this is a very limited test as well. This is an AB test just on 10 tags and for a period of two weeks. And what we're really looking for here is it's kind of hygiene, right? We've got some kind of qual feedback which we're picking up from readers. But typically the main thing we want to check is is this doing something bad? Is it actually making click through worse? I put a lot of money on it not doing that, but. But that's basically the level we're at the moment is the just a very simple hygiene check. And what we will probably do is after talking through what we've seen in this process is in a couple of months time, probably run it at greater scale across a greater set of tags in a more open way and again at that point collect a lot of user feedback qual about exactly how helpful they found it.
- [25:41]
Kamika McCoy
Right. But the approach that you all have seems really thought out in terms of limiting it to human ridden, human edited headlines and then also having human powered reinforcement learning as part of the process.
- [25:55]
Chris Moran
It's not exactly reinforcement learning. The machine isn't learning as it goes along. But at the very least, what we're actively trying to do is build that qual feedback directly into the technical aspects of it to try and make sure as best as possible that it's doing a good job. Again, that is not a guarantee. I mean, most people have related content algorithms which can throw up unfortunate comparisons. But part of this is us understanding how comfortable we can get with some level of risk and whether or not we can guardrail it enough. Just to be absolutely clear, there are two other aspects to this which are important. We obviously have a very large red button which says turn this off and when you're testing 10 of those, you can do that. We're never going to turn this on on 27,000 tag pages. And there are certain kinds of tag pages we're just going to avoid. So people, for example, this kind of approach unfortunately is quite good at kind of creating what our Editorial Legal team would call rogues galleries, where it finds one person who's done something wrong and then it finds five others and buckets them all together. It's not ideal and brings risk, so we almost certainly won't turn it on on that. So again, there's a whole series of conversations with Editorial Legal and others about just whether or not we turn it on on a given kind of thing. And the other thing that I think is super important is this is not being generated for an individual user. At the point the user arrives. It's been generated on a kind of, at the moment, a kind of 48 hour kind of flip switch. But really what it will do is when it sees a particular tag has a large amount of content, it will regenerate, but that will be the same. Same thing for everyone. Which again.
- [27:36]
Kamika McCoy
And does that get flagged to an editor?
- [27:39]
Chris Moran
Exactly that. So we'll have a thing which will say, look, this has been newly generated. And I suspect the next step will probably be some kind of automated system that also looks for any risk when something's been newly generated and might flag that to people for people to check. Again. That is not perfect. But it's also important again to say if we roll this out for another test, we're still going to be rolling out on a corpus of tags where we feel we can have control of that in the first instance.
- [28:10]
Kamika McCoy
Well, it's really interesting stuff, Chris. Unfortunately we're out of time, but feel
- [28:14]
Tim Peterson
like could talk to you about this
- [28:15]
Kamika McCoy
for hours, though we still have a few more days here in Vail. So Chris, thanks so much for taking the time.
- [28:20]
Chris Moran
It's a pleasure. Thank you.
- [28:31]
Tim Peterson
Thanks for listening to this episode of the Digiday podcast. If you enjoyed it, please leave us a rating and a review on Apple Podcasts, Spotify or wherever you're listening. Get more from Digiday with our daily newsletter sent out each weekday morning. Visit digiday.comnewsletters to sign up.