Loading summary
Transcript34 lines
- [00:01]
A
Your film is now ready to be shown.
- [00:12]
B
Good morning. I'm Justin Hendricks, editor of Tech Policy Press. We publish news, analysis and perspectives on issues at the intersection of tech and democracy. Fantacion Maldita is a Spanish nonprofit that has been working on information integrity and fact checking since 2017. Based in Madrid, the organization has expanded well beyond its origins in verification journalism to take on platform accountability and investigative research into the mechanics of disinformation. One of the most recent investigations out of the organization focuses on TikTok and what they found raises pointed questions about the platform's creator monetization program. Researchers at Maldita documented a network of hundreds of accounts spanning 18 countries that were producing AI generated videos of protests that never happened and, and doing so not out of any discernible political motive, but to accumulate followers, qualify for TikTok's revenue sharing program and in some cases sell the accounts outright. Let's jump right in.
- [01:08]
C
My name is Carlos Hernandez Echoarria. I am the associate director of Foundation Maldita Puntos, based in Spain. I do the public policy work there, platform accountability, et cetera.
- [01:19]
D
Hi, my name is Marina sa Cristan Hidalgo and I work as a public policy officer here in Maldita and my background is in international relations.
- [01:28]
A
Carlos, just quickly, for our listeners, anybody that's not familiar with Maldita and what it gets up to, just explain what's your line of work, what business are you in?
- [01:37]
C
So Maldita is a nonprofit, is a registered foundation. We have been, you know, working since 2017. Our mission is fighting for information integrity and against disinformation. We started, you know, basically doing fact checking, but quickly understood that we needed to do many other things, including education, engineering, and particularly public policy, in order to get the impact we were pursuing. And in that line, we are very much convinced that the role that the big deal platforms play on this information is so crucial that we need to be there and investigate them and see whether or not what they say and their legal obligations align with what they do and, you know, how they comply with laws.
- [02:26]
A
So I'm excited to talk to you about this. A new report that you have out on TikTok and polarization. You state that TikTok is financing polarization in Europe and elsewhere, and we'll talk a little bit about what you mean by that. But Marina, can I just maybe ask you to explain your methodology and investigation process for this report?
- [02:49]
D
So once we started with the investigation, we really didn't know the scale that it had. While we were looking to write on a Different piece on a different topic for TikTok, we came across these few examples of protests supposedly taking place in Spain, but we could detect that they were AI generated. So after that we found some other accounts with similar names in the profile of TikTok accounts. And there were protest in Italy and Germany and Great Britain. So then we started realizing that this was not one single campaign, but more of a modus operandi. Right. So there was something behind it and the reason why people were doing it. So we started trying to collect more of those to see what countries could have been impacted by, by this kind of content, which is AI generated protests that didn't happen or that were not real content of existing protests. So we started annotating all of them to see kind of the impact and tried to investigate what the reason could be for them to, I don't know, play along with this type of content.
- [04:06]
A
So give me a little more detail on your investigative tactics, like what tools do you use, how do you go about this?
- [04:13]
D
So it really depends on the investigation, but for us, it's very important to have everything sort out and organized in a specific manner so it is easier for us to assess. For example, coordination. Right. For this investigation, I think that we came across many different campaigns, many different individuals managing different accounts. But since we tried to collect all the accounts that we detected in a very systematic way, we could see these small details on usernames, on creation date, on name changes between the accounts that we were able to notice only because we had collected the IDs. So all these things gave us all the knowledge to assess coordination and the scale of the issue.
- [05:06]
A
And as far as identifying AI generated content, I mean, the forensics is difficult, hard to tell often what is AI generated, what is not. It's always a cat and mouse game between creators and forensic tools. How do, how do you go about making those types of assessments?
- [05:23]
D
So this is a very specific case for this investigation because some of the creators, let's call them, they didn't really care that you knew it was AI, so they were just putting out the content. So sometimes they will leave the AI watermarks for Sora or for Gemini's creation generator of videos. So that was pretty obvious just to look at the watermarks. Sometimes even TikTok had those labels marking that it was AI, but this becomes a problem later. So even if people looking at the video in the platform know isai, we also have people taking that content and posting it again on a different platform. And that label is missing now. But yeah, the assessment was Pretty easy, let's say, for it to be AI. Sometimes we could even see details of that watermark being blurred on purpose. So you could see in a corner that there was something moving differently where the sort of watermark would exist. For example, we still saw a lot of mistakes of the AI, so one arm going through the girls, person that was speaking or details like that. But we also used some softwares which are not perfect, but at least they could give us some indication that in fact, and by the end of the investigation, after 5,000 videos, I think we're very well trained now to know even the colors of the type of content or the movement that they usually generate.
- [07:02]
A
Talk to me a little bit about scale in terms of what you were able to uncover. You've got lots of connected accounts that appear to be coordinating in some way. What was the scale in terms of what you were able to patch together?
- [07:15]
D
So this was a collection of around two months that we dedicated to looking for new accounts. And we ended up with 550 accounts that we're targeting. Let's say they were focused on 18 different countries. So the scale was pretty big because they were producing videos at scale. We collected over 5,800 videos that all had the same characteristics. While we were doing this, we also came across different types of content. But for this investigation we wanted to have two basic characteristics. So AI generated content and videos that had to do with protests because we also came across videos of snow in Russia in amounts that they were not possible. Also heats in Gaza, in Palestine, which was also civic discourse content in a sense, but not in a protest way. So we also came across different types of content, but those are the two characteristics we were looking for. But yeah, the scale was big, but at the same time, it was not an exhaustive investigation. It was not our intention to gather all the accounts. They were doing this just to publicly report on an issue that the platform has.
- [08:39]
A
Let's talk a little bit about the business model. I mean, I think phenomena similar to this have been observed, you know, across the world, across many different social networks and thinking even about reporting in particular by, I remember Karen Howe and MIT Technology Review years ago, looking at the way that creators were taking advantage of conflict driven videos and political content to exploit Facebook's advertising platform. What did you observe here about how, you know, these creators are taking advantage of TikTok's Creators Program?
- [09:16]
C
The most interesting part for me in the investigation was precisely the role business models play on this. You know, beyond the fact that there is so much, you know, Political disinformation that just couldn't exist without this economic incentive by the platform. I think for me, having a look at the industry that has been created just because these creator power programs are not that don't have the safeguards in place that they needed to have it was amazing because Marina can describe in detail how these people, they weren't shy about volunteering information to us in the sense of this is how I do. I use several email accounts that I open and in that way if I use a vpn, I can use Sora without restrictions in Canada and then I can grow an account to this point. Then if my VPN is pointing to a UK server, I can get them monetized. So they were very clear first of all that they weren't into this kind of content per se. It's just that they found that it's the content that TikTok's algorithm wants and rewards and then that they were able to run the same scheme time and time again. Like, you know, many of these people were probably running not only several accounts but also some of them, you know, with an eye of selling them online, some others just to keep producing content to be monetized. So it's, it's such a clear indicator of a problem that would never occur if not for the failed policies of a platform. So for me it, it's, it's a great, great, great demonstration of, of how platform policies, when they are not applied, they can not only not solve the problems we have been pointing at for years, but actually create new vulnerabilities and make our democracies less safe just because they are funding the same things that they are supposed to be fighting against.
- [11:29]
A
So we've got a group of people who are not necessarily ideologically motivated, they're financially motivated, they're sharing content across the political spectrum, you know, don't appear to, you know, have a side per se, simply interested in sharing conflict material, you know, benefiting from it. Talk to me a little bit about, you mentioned VPNs. I mean this idea of bypassing geographic restrictions on tools on the creators program itself. What's that about?
- [11:56]
D
One of the creators that we talked to that was behind several of these accounts, so around six of them. He explained the whole process to us because he was interested in us collaborating with him. But what he said that in the first place he usually uses a VPN only to be able to use Sora. So he's located through a VPN in Canada only to be able to generate those videos and then he creates accounts in those countries that are part of the monetizing program in TikToks. Right? So it's only United Kingdom, the US, Mexico, France, Germany, South Korea, and I might be missing one of them, but it's not many countries. So you really need to create an account that is based in one of those countries to be able to monetize after you reach a threshold of 10k followers. Right? So we know that something is happening there. Even if the program is only located in a few countries, they are being able to access the program anyhow. And it's not only those two characteristics that you are located in one of those countries and that you surpass the 10k followers threshold. You also need to not be against community guidelines so you don't have one violation. And that is also something that we think they are doing because they are producing AI generated content that affects public discourse which is against the community guidelines. Right? So if the enforcement of that specific line in the community guidelines was actually taking place, they wouldn't even be able to monetize the content afterwards.
- [13:50]
A
And did you have any interaction with these creators?
- [13:53]
D
Yes, because as you said, we noticed that the point of making those videos was not ideological or otherwise. They were not very successful in it because of how much they used to change the topic, go from one thing to other. But also not even political content. Sometimes they would just start posting videos of kittens, then psychics, and then all of the sudden protests. Right? So we were thinking the agenda for these people is pretty weird. So that's why we noticed that it must be something else that was happening. So then we started after we collected many of them, we saw some of them that in their bios stated, if you want to buy a US based TikTok account, DM me or please follow me. My dream is to reach 10k followers, which is, as we know, the threshold that TikToks requires to join their program. So that's when we noticed that the point was not ideological or political, but it was to make money. So we reached some of the accounts that had in the bio, the selling accounts announcement, and they of course told us that they were in fact selling different accounts in different locations, but also that they were already monetizing. So it is more valuable, let's say, to sell an account that has good engagement metrics, that has a lot of followers and even on top of that is already part of the monetization program. Right. But we also came across was this one person that I already mentioned that he willingly told us the whole process, from the creation of TikTok accounts to generating a content about the news, because that's what he said works best, or gaining a lot of views and a lot of engagement very quick to reach a lot of followers very, very quickly. Right. And then he said what topics he think he thought that were working and which ones were not, for example. So that was very interesting, and that was a very valuable testimony that we had from him.
- [16:14]
A
You already mentioned kind of some of the aspects of the policy context on TikTok itself. You know, the extent to which some of this behavior might, you know, offend TikTok's own guidelines. Talk a little more about that. And then I want to ask you also, of course, about the policy context in the European Union under the Digital Services act and what the implications for this type of behavior could be.
- [16:40]
C
Yeah, I think it's very interesting and it's something that. It happens to me, particularly on TikTok, because I need to follow the platform policies about misinformation very closely. And I find that they do have great policies that they keep updated and that they are generally strong to deal with the kind of challenges they have on their platform. The problem being that they are seldom enforced or they don't have the capacity or the will or I don't want to characterize that. But the policies are not the problem in this case. I think TikTok has made the right call in making sure that they don't have kind of a blanket policy against AI generated content, but draw a clear distinction saying that, you know, you cannot do this in issues that are affecting, you know, public opinion in a significant way, and, you know, that coverage of current events need to be authentic in that sense. And I think, you know, it's. It's very important that in the. As these tools become more and more powerful, this is the right policy to follow. Like, if you have a policy against generative AI content that might look like it's being genuine and you are using in this area of public interest, that's the right way to go. But then you need to make sure that you are able to actually spot that content or at least respond when they have this massive number of views that we observe here. I think the Digital Services act in that sense, it provided a very strong notice an action system for illegal content under Article 16. But I think it's also, it was, since it was conceived, this, this European framework, it was already clear that it needed to have this other, you know, this other part of the effort that had to do with making sure that the terms and conditions for the platforms, the internal rules were if they existed, they were available and they were actually enforce. So on that Article 20 of the DSA is pretty clear in allowing users to file complaints but also establishes an obligation for the platform when you know, their rules haven't been followed to, you know, provide remedial act. And I think that is very important because if you look at the article in particular, it doesn't only say when a platform has, was supposed to delete an account and did not, but it also says the other way around. So it's not only about protecting users when there is over enforcement of the policies, also when there is lack of enforcement, which is what we are discussing here. So in the EU regulatory framework, I understand how DSA enforcement so far has been focusing on more like, I don't want to say low hanging fruit, but more crystal clear cases around illegal content and their patterns. But I think the time is going to come for regulators to make sure whatever promise these platforms make to their, to their users is actually fulfilled. I think this is going to be also an interesting area to follow because I think as these platforms are forced more and more to confront the fact that they are not following their own rules, they are going to be starting to relax their own rules and make some of them disappear, which will be an interesting side effect of this whole debate about regulation. But clearly in this case the policy is good. It's just that there is this massive hole in enforcement that you know, it's, it's hard to think like how this, the volume, this, this kind of practice has, has acquired how, how this hasn't been spot by by the platform.
- [20:38]
A
So as you say, I mean, you know, we just saw set of preliminary findings against TikTok in particular around some of the allegedly addictive aspects of, of how the platform operates. And it is certainly, I'm sure much more complicated to look at these sort of systemic risk questions. You know, things like political polarization or disinformation, you know, these are obviously much more difficult to regulate and there are all sorts of speech implications and other questions here. I don't know, I mean, I'm wondering, you know, I guess I'll ask this, you know, the DSA also, you know, contains within it some mechanisms for providing researchers like you with better access to platform data. Would that have mattered here if you know, for instance, the next step is to put in a Data request to TikTok to be able to study this problem at greater scale.
- [21:33]
C
I think it could mean the world, it could make our, not not only our lives easier like Marina and mine. It's, it's about the information that is available over here. For example, we are assuming or, or we are being told by these actors that they need to reach this threshold to monetize on TikTok when there is no real solid reason for TikTok not to be able to tell researchers this account is monetized or not. It doesn't have to involve specifics about payments, though it could be good. I mean, there are many ways to do it. But for example, on the, on the side of monetization, really on any side, but on monetization there is so little data that platforms are being, you know, opening to the, to the researchers that there is this huge driver for the production and dissemination and this information that it's very much understudied because platforms are actively blocking it. So yeah, definitely in the provisions that have to do with that access in the DSA are fundamental for all the other pieces to come into place and to study systemic risks in any meaningful way. Because if not, you end up in a situation like we are seeing now, which is that right now, for example, in the US and you know this very well, there is, you know, part of the litigation against platforms regarding online harms have to do with the fact that their internal teams have been researching on some of this, you know, ugly side of its own company and they have put probably the findings in front of management and management have ignored them. But we are already seeing signs from many of these platforms in which probably the top echelon is now saying, you know, maybe let's not investigate ourselves in this way because that's going to create a paper trail that is going to create us some problems in the future. So either we unblock, particularly in the European Union regulatory space, either we unblock the, the possibilities that DSA brings for research in terms of data access, or we are going to be flying blind really quickly and we need definitive answers on these kind of things. Like how many people saw this stuff is interesting data, how. But whether or not there was algorithmic amplification and there was monetization for real, that is probably the smoking gun in ascertaining whether or not the platform bears legal responsibility. And I think we are, we have gotten a smart at finding ways to prove that that doesn't, that don't rely on the platform's data. But it would be much easier and much more straightforward if we would have that kind of data from them which are the ones who have it.
- [24:27]
A
Are you Looking at other creators programs, other monetization programs at other platforms.
- [24:32]
C
Yeah, we have been looking at not only monetization but also, I would say the financial aspect of it in a, in a broader way. So not only creator programs, but also how advertising is used in this, in this context and how for example, some channels are providing their own banners so you can, you know, basically PayPal them or Stripe them money. So yeah, we are looking at a number of things that we used to have good or you know, the best that are available on, on, on Meta. I think we're all aware of the work by What2Fix this organization that has been tracking this issue of creator problems for quite some time. But we are in the dark about so many other platforms. The data monetization on YouTube, for example, which is for me one of the most crucial vectors of this information in many areas. We are assuming, for example monetization there because we are able to track whether or not the videos have ads and if they have ads, you can basically safely assume that there is, they are, you know, the user is engaged in some kind of revenue sharing program with YouTube. But as the creator programs grow more and more complicated, it's going to be difficult to have meaningful research on this if not by the regulators making sure this data is accessible. And I also really don't see any argument against its own privacy or commercial secrets or that because I think there is already about decades of precedent that say that commercial communications can normally work in a separate legal environment from the normal freedom of speech of users. And when money is involved to promote particular abuse, users are entitled to a higher degree of transparency of, you know, who pays how much, all that. So I think this should be one of these things that are priority one for regulators everywhere.
- [26:47]
A
I suppose my last question is, you know, have you heard any response to this report either from TikTok, from regulators or from other researchers in the field?
- [26:57]
C
Yeah, we are super encouraged about all the colleagues in the, in the, you know, and researchers who have reach out, but particularly about the regulators who want to know more about this because I think something that they have found very interesting in this, in this investigation is yeah, that particular notion that many of the things that we have been suspecting for long, which had to do like, you know, is there a link between business model and this kind of content that it's per se, you know, polarizing, it creates problem in public life, all that. I think this is one of the clearest links that we have seen so far on this particular platform. So yeah, I am very much encouraged. And I know the timings of regulators are quite different from independent researchers like ourselves, but to be fully honest, I don't think we have ever had such a strong response as we have had, you know, our last investigations, particularly on on TikTok policies on this matter.
- [28:10]
D
I think that is also because you mentioned before the articles in the DSA about systemic risks are very vague, probably on purpose because they have to cover a lot of things that sometimes we don't even know yet what they are going to mean. And this example specifically is very obvious. Let's say it's money going from a platform to an individual which is going against the community guidelines. Right. So I think this is why this investigation specifically and this type of scheme that doesn't only affect protests but other types of content that becomes disinformation. It's very interesting for DSA enforcement too, so that to make money on the side of creator programs or ads, it's something that people are interested to look at and platforms are not very interesting to hear about.
- [29:03]
A
Well, I appreciate you two taking the time to walk me through this research and I hope we'll have the chance to talk about additional work in the future.
- [29:12]
C
Thank you very much Justin. Thanks for providing not only us, but all of us kind of this space to discuss these things that matter. I think on that Tech Policy Press is such a great place.
- [29:24]
D
Thank you very much.
- [29:35]
B
That's it for this episode. I hope you send your feedback. You can write to me at justinettechpolicy Press. Thanks to my guests, thanks to my co co founder Brian Jones and thank you for listening.
- [29:55]
D
Tech policy press.