← The Tech Policy Press Podcast
Loading summary
Transcript57 lines
- [00:01]
A
Your film is now ready to be shown.
- [00:12]
B
Good morning. I'm Justin Hendricks, editor of Tech Policy Press. We publish reporting, analysis and perspective on issues at the intersection of tech and democracy. The past two decades have witnessed the emergence of the professional field known as trust and safety. Once a loose collection of ad hoc problem solvers inside early technology platform companies, trust and safety teams gradually evolved into more formalized structures charged with defining, enforcing and explaining the rules that shape online Life. But in 2025, some say the field's best days are behind it. Political headwinds, organizational pressures, and the rise of compliance oriented governance have reshaped the field in ways that many experts and practitioners find troubling. My colleague, Tech Policy Press contributing editor Dean Jackson, recently hosted a discussion with three experts to explore how the trust and safety field arrived at this moment, what forces are pulling it, new directions, and where opportunities for meaningful reform might still lie.
- [01:08]
C
Here's Dean Past few years have featured a great deal of introspection about a professional field which has come to be known as trust and Safety, comprised of the people who develop, oversee and enforce social media policies and community guidelines, Many scholars and advocates describe it as having reached a turning point, mostly for the worst. Here to discuss the evolution of trust and safety, not coincidentally the title of their forthcoming article in the Emory Law Journal are Professors of Law Danielle Keats Citrone and Ari Ezra Waldman. We are also joined by Jeff Allen, who's the chief research officer at the Integrity Institute, a nonprofit whose membership is composed of former industry professionals. Thank you all for being here.
- [01:43]
D
Thank you so much.
- [01:45]
E
Thank you so much.
- [01:46]
C
If I can take the prerogative to flash us back to the 1990s. After reading your paper, I revisited an old essay by Ethan Zuckerman at UMass Amherst on his time developing trust and safety at the web hosting company Tripod, which was a competitor to GeoCities, which saying those words out loud now makes me sound a little bit like an archaeologist. But he describes how those companies encountered novel legal questions as they were thinking about their own trust and safety policies and how they often responded to those through the path of least resistance by building tools on the fly and making ad hoc decisions that really only later became codified into norms and policies. And your account of trust and safety on social media struck me as very similar, even though it takes a later starting point. So I wondered if to start out, if you could tell us just how did the field evolve during its early years?
- [02:36]
D
Well, I first started writing about cyber stalking in 2007 and talking about cyber Mobs and at the time, industry folk. Well, really nascent, you know, these nascent tools like Twitter and then Facebook folks would reach out to me and say, hey, I'm the. I'm the only one at like Del Harvey. I'm the only one at Twitter dealing with spam and abuse. And I need some help because I understand what stalking and harassment feel like from the inside, having been stalked herself. But I want to help the people I work for. She was one of two. She was the only. Basically, she hired one other person to help her. Really. Twitter was focused on spam impersonation and copyright violations. And she's like, I'm seeing a whole lot of abuse on this. Early platforms 2009 early 2009. I hear you've written about cyber stalking in a piece called Cyber Civil Rights. I don't know many law professors. I'm not a lawyer. Dell says to me, but can you help me figure out what I need to say to people in charge to show why a. What online stalking looks like and is and how harmful it is and help me. Can you help me? And I was like, absolutely. It wasn't like no one had ever thought of compensating anyone. I was left, course not. I was like, let me help you, of course. Couldn't be more meaningful. And so it was sort of. That began some of my work with folks in trust and safety, which they didn't understand themselves. They didn't call themselves trust and safety. Right. Like, Del was like, I deal with copyright span and hate. She didn't call herself really anything in that sense. Right. And at the same time, so once I talked to Dell and all these other people would get reach out to me to say, I heard you help Del Harvey. Could you help me? And the more that I wrote about online abuse and different forms of intimate privacy violations and cyber stalking more generally, the more people would reach out and say, will you help me? Then that evolved into helping and advising Facebook, advising other companies to figure out what do we mean by hate speech? That evolved over the top. And so it was largely a discussion of policy. What do we mean by terms that would be in policy statements and then dealt with ex post. We largely like after stuff happened, we had discussions about products, but it was in the early years, we weren't talking about design. Right. It was really like, how do we figure it out when someone, someone, you know, lets us know there's been abuse that is silencing them and scaring them, terrifying them, possibly ruining their lives? And how do we. How do we deal with this. So I would help them define things, give examples. And in 2011, when I wrote a piece about hate speech and intermediaries and how they make they might, what we called them intermediaries, Helen Norton and I, how they might define hate speech and why they should. Why they should be transparent in their policies, why they need to give examples, why they. You go figure out which kind of definition you. You want of the harms you want to avoid. And that then led to much further conversations and work with companies beyond Facebook and Twitter. Does that make sense? So that was, like you said, what are the early years? And what I can tell you is those early years involved discussions about policy, speech, policies and practices, and that those ultimately led into conversational work about product design. But that evolved over time.
- [06:10]
E
The only thing I'll add about the evolution that was absolutely perfect, Danielle, is that I want to disaggregate a little bit the two descriptors that you used in your question, Dean, which was you suggested that early on it was pretty ad hoc and companies took the path of least resistance. Those don't necessarily have to go together. Danielle described a world that was very ad hoc, where someone like Del Harvey was, saw a problem and then figured had to figure out from scratch how to deal with it. And after people like Danielle, led by Danielle really, and others who would engage with them, defining these terms and also creating these transparent policies. But at the same time, these platforms were growing, and when you had to start doing these things at scale, then you start getting more professionalized policies. So if the story that Danielle is telling of the beginning of, of trust and safety is very ad hoc, there was no such thing as trust and safety. And then all of a sudden you add in content, moderation at scale, and other problems that a more professionalized company will have, then you start getting policies, organizational charts and reporting structures and more defined rules that are now, instead of a thousand words, they're 25,000 words. And you get contract hirees that contract workers who are doing this kind of work and making these kind of decisions at the moment. So over time, and we're really not talking that many years, over time, it moves from an ad hoc process where it's one person with Danielle in a room, to conversations and then to systems, but the question of whether they enacted rules or followed rules and that were characterized by the path of least resistance, that's an entirely different question that I hope you return to.
- [08:07]
C
Could you put a rough date from the beginning of this work in, say, 2007, when you first engaged Twitter, Danielle, to like, when things started to become more formalized. Because your piece, after all, goes all the way to the present day. And so I'm wondering where in the timeline you've just landed us.
- [08:24]
D
Okay, So I would say that, and I don't know, Jeff, if you're going to disagree, but at least for my own, and this is just anecdotal, this is me, Marianne Franks, the Cyber Civil Rights Initiative, which we established in 2013. But we long did work before we established CCRI, either individually or with Dr. Franks, working with companies on this kind of ad hoc, in the sense of less professionalized basis, that lasted, I would say, from 2009 to about 2011 and 12. And during that time period, and I never forget moments like the leaking of Facebook's internal employee manual on hate speech that I remember sitting in a room at Hogan Levels with Kevin Bankston and this leak had just happened. 2011, the leak of their terms of service. They were not transparent about all this policy discussions I was having with them.
- [09:22]
A
Right.
- [09:22]
D
It was all just internal to the company. They were trying. They were working hard on figuring out what is stalking, what are threats, what is bullying mean? Bullying different from stalking. Right. Like all these things they were working on, they weren't telling the public. Because section 2 30. Right. Is really important to note that there's law that shields all of these platforms from responsibility. So they were making. And this goes back to Ari's really important point is that it wasn't like they were shy about doing it. They were actually pretty proactive given the fact that they enjoyed legal immunity. Right. Especially when they were small and we weren't talking about big advertising dollars. So, you know, these companies, I would say between 2009, 2011, 12, are writing policies, they're working on harmful activities and thinking about how they would deal with them. Right. Sometimes more blunt than we have now, the kinds of tools, but nonetheless. And the pressure started arising when there would be leak from people, as Ari was talking about. Increasingly you have contract people executing these policies and they sometimes would talk to the press and leak these policies. So it's such a different world where you go and you click on and you go to terms of service and community guidelines and you see these very long explanations. They did not exist at all. You know, Twitter might have said five words in the beginning. We do not allow copyright violations, spam impersonation and csam, period. The end, like no other explanation. But internally, sorry was our explaining. They're working on it on the inside. Right. Defining Things trying to be more systematic, growing into policies and systems that professionalize it, it least to a certain extent. And it's really like, at least from what I saw from the outside and inside 2011, 2012, 13, we start to see not only leaking of policies, but more transparency because there's pressure to have more transparency and people are writing about it. I'm so not the only law professor. Like there are tons of us now thinking and writing about content moderation, stalking, harassment, harass, non consensual intimate imagery, as I was doing with Dr. Franks, right? And so I think it's in that time period and just before Gamergate, just before the fappening, where I had been watching women and minority sexual and gender minorities being stalked, harassed, threatened on Facebook and on Twitter and other sites in an awful persistent way, right? And they started to work on it because they wanted to work on it, but it wasn't clear to the public they were doing it. And it was really only in 2014. And I think, and Ari, you can help me here. But it's only like in 2014 where we get high profile abuse of high profile people, women in gaming, you know, Zoe Quinn, Brianna Wu, other wonderful humans that I know and then very high profile celebrity that catch the media's attention and then force a PR in some sense reckoning and an advertising reckoning. So it wasn't just remember, law shields these people from responsibility, all these platforms, but advertisers, my own experience with these companies started saying, you know what, it's not cute to have Toyota ads run against rape threats targeted against Brown and woe and doxing like her home address, et cetera and calls to rape her, like, I don't think I'm into that, right? And there were other advertisers who said the same and you saw and then the AG of California, Kamala Harris got involved with non consensual incident imagery. And it's like around 2014 that we see far more funds, resources, you know, dedicated to content moderation, far more formalized policies that they then post and share online. And that has a lot to do with advertising pressure. I always like to think they did it for the goodness of their heart that I was. And Marian Rex, we are so convincing about the harms. But I think in truth it, of course it was market pressures that actually were brought to the fore with our pressure that helped. But it was really Jennifer Lawrence, God bless her writing in Vanity Fair, the posting of my nude photo is a sex crime, right? And it's Brianna Wu talking To the press. The Boston Globe, Washington Post, Zoe Quinn, right. Anita Sarkeesian, who'd been tormented for years a woman in gaming and writing about, like, the gendered misogyny in gaming. I think it's then we had a reckoning of professionalization.
- [14:07]
C
I remember being on a city bus in D.C. checking my Twitter, and I followed, like, a fair number of the gaming press, just as a hobbyist, and it was like my feed had been invaded. I had no idea, no context for what was happening until I. Until it started to get news coverage. But I saw it happen in real time and never would have predicted the sort of continuing shadow that those trends cast. And I'm glad that you've raised them in the context of this conversation and the origins of this work. But I want to make sure we get Jeff in here because, Jeff, you were a data scientist at Meta. Correct me if I've got your dates wrong, but from 2016-19, which puts you a little after the period that Danielle just described. But I'm wondering about your reaction to the way they've characterized the period of trust and safety that maybe came right before your entry into that field, but also, you know, in general, the way the piece struck you. I'd love to get your perspective.
- [14:59]
A
Trust and safety is such an interesting field because it is, in part, so old and but at the same time so young. And like, literally the term of trust and safety dates back to, like, 2000. And I want to say ebay was the company where trust and safety was first coined. And it makes sense that ebay would be one of the earlier companies worrying about this because people didn't want to buy a product and then never have it shipped to them on ebay. Right. That's a trust issue. Right. Like, sale of illegal goods is something that ebay had to be concerned about. There's a safety issue. And so, you know, it sort of feels natural that ebay would be an early sort of thinking in this space. But I do, I do think that, like, they. They hit a lot of the right buckets, like Danielle and Ari. Like, you get it mostly right from my personal experience, where, like, from the outside, it probably feels very reactionary. But usually what's happening is that there are people internally that care about a particular issue, and they can't get attention within the company until something external happens. And then once the external thing happens is when the company actually takes movements on it. And so I think a lot of these issues that we've seen have, like, their precursor in, like, a couple of Employees who are like, hey, this is a problem, we should deal with it and then sort of like advocating for that internally until finally there's an opportune moment in the public when they can be like, hey everyone, do you remember that thing that we've been yelling about for a year here it is getting us negative press out in the public. Can we please do something about it now? You know, it feels very like, like I said, it feels very reactionary from the outside. And it's easy, I think, to be kind of cynical from the outside and be like, ah, like the companies are only doing something when they're getting negative press or when advertisers are complaining or something like that. And I could definitely understand that sentiment. Right. But there is, there is some foresight that goes into it and there are people that are proactively thinking about it. I like to point sort of the first trust and safety or integrity thinking, first example of that in the public was actually Larry Page and Sergey Brin in the Page rank article from 1997 or 98 or whenever. And they talk about like in the future being number one. Ranking number one on a search engine is going to be really, really valuable. And so if you're running a search engine, you really have to worry about bad actors who will try to figure out how to manipulate the system to get to that number one spot because that will have much, a lot of value from a marketing standpoint. And they were concerned about it from a marketing standpoint, which I think like is a lot of the activity. But it's certainly not the only ones. Right. And so they're talking about that in 1998. You know, fast forward to 2005 when you know, one of the first search engine integrity issues comes about. Political ones at least, which was the miserable failure Google. Right. And that's where like a bunch of bloggers came together and they took the phrase miserable failure as their anchor text and they linked it to George W. Bush's biography on the whitehouse.gov website. So if you Googled miserable failure, the number one link was George W. Bush's biography on the White House, which is like kind of one of the first examples of a political influence operation campaign being run online by some definitions. And that was, you know, you know, Paige and Brin, you know, sort of predicted that kind of stuff to a certain example. I think another space to really consider about is ads and ad integrity because, you know, while they're, while 230 broadly protected them from any liability around sort of the organic Content that was being uploaded on their platforms and they're distributing. They were not protected around ads. And especially all around the world there is tons and tons of regulation around ads and what kind of ads are and aren't allowed. And so like ads has been another place where like the trust and safety has been kind of like a long standing thing. We just didn't call it trust and safety, we just didn't call it like, like the professional name. But it was sort of like, oh yeah, ADD Compliance team, you know, and it's their job to make sure that there are no firearm sales in the uk, there's no prescription drug sales in the eu. Right. You know, there's tons of laws around the world that the platforms have to comply with. And so they did build up that infrastructure quite early. But then I think, you know, from Danielle, from like the, the, the sort of like 2011, 2012 era onwards, you know, the thing that's getting really attention is societal level impacts. Right? So like there was the Arab Spring, which was a mix of really positive stories and really negative stories. Like that was around, you know, 2010 to 2012. There was ISIS and terror, you know, terror groups, you know, using the platforms, which kind of takes you into 2014. Then you're into Malaysia, Cambridge Analytica, you know, the Russian IRA operation, 2016, and then you're off to the races where it's like, you know, it really feels from the outside like the platforms are on the back foot and it's just like new, new exploit, new vulnerability, new set of bad actors that are using the platform to do something bad, you know, after another. And the platform is responding to it. And my personal story at Facebook, so Yeah, starting in 2016, ending on 2019, you know, my integrity time started kind of late 2017, early 2018 is when I was on integrity projects. And you know, coming out of, you know, Malaysia, the IRA scandal and Cambridge Analytica, there was kind of a beautiful moment inside of Facebook when they're like, you know what everyone, we're going to take this really seriously. Like, every product team, you're all going to spin up your own integrity, you know, versions of it. So my first Integrity team was the Pages Integrity team, which was based off the Pages. Org. And they're like, hey, like, you know, the IRA just used the Pages product to reach, you know, over 100 million Americans. We need to take this, we need to seriously consider what are some bad outcomes that might be happening on Pages. And the company really took it seriously. It was honestly kind of amazing to Watch like as every different product team spun up integrity teams that were worried about any negative impacts coming from them and taking that seriously. And yeah, it was great. I think it was great for a while. I think we're seeing like I don't think that trust and safety is there, but we are entering a new era for sure.
- [20:52]
D
Can I just intervene just to I think add a little color? Because we're not going to disagree, Jeff, about the people who really moved the needle, who really cared. So whether that was Yonatan Zenger and Leah Kirstner at Google, whether it was these are people I worked with really early on, you know, whether it was Dave Wilner at Facebook, whether it was Dell Harvey and then ultimately Vijaya Gadda at Twitter. Right. They made a difference. Or Sarah Hoyle at Twitter 2014. Right. But to be clear, yes, Google, they know as a marketing matter Justin's safety in their Page Rank article. Right. But I have to say even though I had, you know, I gave my book hate crimes in Cyberspace at Google to a really big audience 2014 about cyber stalking and I had internal advocates for addressing non consensual pornography and then called revenge porn and cyber stalking. But the hardest actor to get on board to address non potential intimate imagery and cyber stalking was Google. The hardest YouTube in meetings to say we don't touch search, we don't do that. Which is was absolutely bullshit and excuse me and not true. Right. We never touch search. Untrue. Right. We going to de emphasize mugshots.com but like could care less about non potential intimate imagery. So the monetary engine that's driving all this led to a very strike oriented at some platforms including Twitter and Google like nope, we're not doing anything. And that wasn't because Del Harvey didn't care and it wasn't because Leah and Yonatan didn't care. Oh, they care. They had me, they read my books, they had handed out copies to a hundred people. Right. They care. But it was the economic model that you know, prevent at least the C suite, as I was told C suites have not having it. Right. And the advertising is not in the United States, Right. Advertising is covered by 2:30. The argument from Google is so two faced. It's my speech, it's not my speech. Right. Like depending on whatever the litigation is. And so in this era of 2009-13, right. These companies are saying oh, advertising, that's someone else's speech were covered by 2:30. And they made the very same arguments with regard to civil rights law and it's sort of leveraging up until two years ago, like literally. So it's absolutely not true in the United States. So230 yes, advertising regulated and hate speech in the EU in countries outside of the EU as these this goes to Ari's point. As these platforms start to really become they are geopolitical, they are across their way behind the United States. Right. Like their biggest users are in India. Their biggest audiences are outside the United States, without question. But still in the United States. Right. They are dealing with issues very differently because of the business of lights.
- [24:03]
C
Cliff shares, you've kind of walked me into my next question really nicely, which is about the sort of role that economic incentives and trends play in this story. So just a moment ago, one of you said that this isn't the End of Trust and Safety, which is the name of yet another article about the state of trust and safety. I also throw in as a reference here, Kate Clonic's the End of the Golden Age of Trust and Safety to talk about the era, Jeff, that you were describing, in which there was this window of opportunity. And it sounds quite normal to me to talk about the sort of triangulation of external advocates, internal advocates, and key decision makers within companies. That's a very reasonable sort of strategic lens to view these conversations through. But the piece that we're all here to reflect on, I thought was really critical of the perception that economic trends had driven changes to the trust and safety industry, including mass layoffs over the last few years. The rollback and you may not explicitly blame economics for all of these, but just to list some of the trends, the rollback, fact checking programs, replacement of procedures like human rights impact assessments with artificial intelligence systems and like, when you look at all of these things together, the message seems clear that for companies at least, trust and safety is viewed as more of a cost center than an investment. And yet there's more than economics and dollars and cents to the place that trust and safety sort of ends up today. So what are the concerns you have about the evolution of trust and safety? Where is it? Where do you find it in 2025? And what worries you? What trends worry you about the space?
- [25:33]
E
There is a role that economics dollars and cents play, and we don't say otherwise. Danielle, Neither Danielle nor I, whether in our piece or elsewhere, never do we say that there is no role that economics and the finance team plays in the evolution of trust and safety. But to suggest, as many other scholars and commentators have suggested, that the reason, and I'll give them the benefit of the doubt and say the primary reason that trust and safety is changing or has died or we have passed the golden age is simply a matter of it's a cost center and now there isn't a lot of money is insufficient. It's not wrong, it's part of the story, but it's incomplete and it's a pretty superficial story if you listen to what both Jeff and Danielle were talking about. I mean, I can quibble a little with Jeff's characterization of what constitutes trust and safety. And then the value of these, quote, putting up teams and all these products. These are organizational stories. These are, these are questions about how different parts of an organization learn and respond to outside external or exogenous and internal or endogenous stimuli or challenges. So to suggest that this was only because of money ignores the fact that organizations operate in certain ways and organizations operate in ways that are independent of money. They operate through how they've organized reporting structures. They've operate through how they define what is and what is not in this department. Meaning that if I say something is part of the IT department, then the IT people are in charge of my budget. But if I split a department's budget through four different other departments, then I'm going begging for four different people to try to get money for what I have to do. That has nothing to do with economics, right? Everyone's budget gets cut at the same time. That has to do with organizational structures. So what was great about working with Danielle on this paper was that we come at this from Danielle's extreme, extraordinary, unmatched perspective on the inside of doing this work. And I came at this from being inside these companies and just learning how they operate and putting that together. So when we talk, and I'll give one example, and then I'll stop talking. So Jeff talked about this really wonderful moment inside a company where something happens and then all of these different products put up teams and then everyone is talking about integrity. And let's for the moment assume that all of these things are under this topic of trust and safety. Although I have normative reasons to want to resist that. Too copious of a definition, too capacious of a definition, but let's assume that for the moment. That sounds great, but the organizational story reminds us that any team can set up whatever team they want, right? I can set up whatever kind of team. I can set up the most aggressive, the most design oriented, the most structural reform teams. Think about Congress. Congress can set up whatever kind of committees that it wants, right? But if nothing happens with Those teams. If those teams are de skilled, or if those teams are ignored, or if those teams are given structural. Structural position, but no internal, structural way to access power, then what's the point? It's almost. Almost. It is a tool of people in power to allow sub departments to set up whatever kind of teams they want to make them feel like they have a voice, but then to disempower them, to show, to make sure that that voice never does anything or never has any or never results in any. In any success or any final determination that affects the business. Right. So we have to move beyond just the economic story to understand the organizational story.
- [29:32]
C
The period from 2018 to 2020 seems like an aberration. Just because of my own work looking at especially Facebook's response to social issues in the United States during that period, it would be hard for me to say that those internal organizational teams didn't have impact and didn't successfully advocate for even sometimes algorithmic or design changes. And yet we've ended up five years after the 2020 election, I think in a place that is much worse and where those teams do seem disempowered and perhaps deskilled. And so, Jeff, if you want to react to Ari, but also maybe give your account of the trajectory of the field over the last five years and why it is where it is, I think that'd be really helpful.
- [30:13]
A
I largely agree with Ari, and I can even say that, like, the foundation of the Integrity Institute owes itself to a lot of those impacts and effects that he's talking about. Right. So, yeah, 2018, 2019, let a thousand flowers bloom. It's a beautiful moment. Early 2021, you know, Facebook is reorgang the civic integrity team, you know, and a lot of those people ended up being like, there's no integrity roles for you. And something like half of the people that were working on civic integrity were like, yeah, there's no more headcount for you to work in integrity periods. You need to find something else to do at this company. And that led to enough disgruntled integrity workers to get, you know, a handful of us to be like, okay, we need organizations in the public that are, you know, saying the things that, you know, we want to be, you know, we want to have said that are saying the things that will help win. Help us win more fights from the inside. So, absolutely, it was a great moment. There was a whole lot of activity around it. And Facebook does deserve credit for at least creating that moment and allowing people to explore these topics. But at the Same time, we weren't able to have the impact that we needed to have to actually ensure that these problems were solved comprehensively. And a lot of those teams got downsized or know, reorged to the point where, like, there's plenty of people at Facebook that were like, okay, cool, we need to think about what is the solution outside of the company to these problems that we're trying to fix and how do we start playing a role in those spaces as well, you know, and so I think we have seen a change. I think if you look at the layoffs and the downsizing of the companies, I don't have a whole lot of data or intelligence that's telling me that, like, that hit trust and safety roles especially hard. Except for like a couple of examples. I do believe that user research teams, particularly Meta, got hit really hard. And then partnership teams also at Meta, got hit really hard. But apart from that, it's like, as far as I can tell, it was kind of like a clean across the board. Every different part of the company is having layoffs, but the user research ones and the partnership teams, that is kind of telling. And how the companies are thinking differently about this space now because user researchers were the primary. That was the primary role for people that were talking to users, doing surveys, doing focus groups on, like, describe your bad experience of the product to me, help help us understand the bad experience better. And as far as I can tell, those are kind of the teams that got hit particularly hard in the layoffs, which means that we're seeing the companies are no longer interested in being proactive and asking the question, how are you having a bad time on our platform users let us know, what bad experiences are you having, help us understand your bad experiences better. And I think that's kind of backed up and further strengthened by reporting we've seen in the Washington Post where the legal teams are putting really heavy restrictions on user researchers and how they can ask the questions and what questions they're allowed to ask and those kinds of limits. I do think there's really something to be concerned about there.
- [33:07]
C
I've seen some examples of some of those user surveys, and I think people really don't know how much time and effort Meta in particular, at least did spend in the past surveying users on the experiences they had on the service. And it strikes me that the results of those surveys totally contradict the sort of move to free speech narrative that the company has leaned into in recent months as, at least as a user desired phenomenon. Right. People largely report that they want to see less of what we would describe as harmful content on the service that they think they're not doing enough to keep hate speech and other things out of their feeds. They want more content moderation, maybe more transparent content moderation. But. And I think you're. That's really interesting. I didn't realize that those teams had taken a disproportionate hit in the layoffs.
- [33:55]
D
Just one quick fun podcast to listen to. It's. It's called Sudir Breaks the Internet. So it's Two Deer's podcast, like 2021. My buddy's a sociologist working at Facebook and you know, he and his team were coming up with ways to how to de. Escalate abusive activity and having. Thinking about ways to have like teaching moments with users and when meeting with Mark and others to kind of present this actually is really effective. When we tell users what they did was wrong and they stopped doing it. Right. The response was that's too expensive. Like, we're not doing that. You have really interesting research teams come and then he basically quits. Like, you know, he and his friends quit. They're like, this is dumb. No one is listening to me. I'm doing this hard work. So even in the golden age, which I'm with you, Jeff. Like working with these companies on Twitter's Trust and Safety Task Force, Facebook, then Facebook non conceptual intimate imagery group, which met very regularly. And Antigone Davis is amazing. Right. You know, all these people who really give a rip. Right. But it's when it's low cost, often when it can be easily systematized. Right. When it's not going to attract, oh, this is too expensive. Right. Then we're going to go to. It doesn't mean that. I'm saying economics just heated an agreement, of course with my, with my co author.
- [35:19]
E
Right.
- [35:20]
D
But it is, it's like that combination of economics and structures and well, if this is going to cost too much, we're getting rid of that structure.
- [35:29]
E
Right.
- [35:30]
D
Or we're not going to pursue this teaching people that they shouldn't misbehave even though it. It will redound to enormous benefits, the prevention of mischief later, societal harm averted. Right. It's just too expensive to have people sit down and talk to people, right. About what they were doing wrong and just to explain the rules. So it kind of gives you a sense of the kind of deeper commitments, value commitments behind that. Like as we saw with Facebook Live, like Facebook Live gets rolled out and immediately two weeks later, like live rape, you know, rapes live streamed across the United States, Sweden, other countries. And Zuckerberg's response is, rather than hitting the brakes on anything, some costs are there, like, nope, we're just going to throw it. We're going to hire 3,000 more content moderators, right? Like, we've spent the money. We're not going to scale back. And that doesn't help. Right. For Facebook Live, which then enables Christchurch Massacre being livestream. Does that make sense? Like they just, what do they say? It's, you know, you move fast and break everybody. Right? You roll out these, right. These products. You don't listen to the insiders or you don't ask at the high level. Roll them out. And then you say, let's attack on safety afterwards. And sometimes as a matter of design, Jeff, you know better than me, sometimes as a matter of design, you can do something. But once stuff is out there, it's very hard. It's often like exposed and that's not helpful.
- [37:03]
C
I want to ask one question about compliance culture and then I'll try to bundle my last couple questions together, which will be about politics and recommendations and whether or not we should be pessimists or optimists about this field. But you know, you're, when I was reading your piece, it reminded me of another piece by Daphne Keller at Stanford on the rise of the compliance speech platform. And you've just talked about the way in which, like, companies sort of search for kind of low cost ways and structures in which they can take action when they need to, but not necessarily as much action as internal or external advocates want. And you write that proceduralist compliance is the dominant form of governance practice in neoliberalism. Right. That when regulations like the Digital Service act in Europe, where a variety of proposed or passed laws in the United States proceduralize trust and safety and make it into a reporting function, this could have a negative impact on the field, that it could replace real protections with forms of paperwork. And when I read this, it's a common critique and one I've seen before and I think a really real risk, one people are complaining about. But I always find myself asking, why is compliance the ceiling for corporate responsibility instead of a floor?
- [38:14]
A
Why?
- [38:14]
C
If there was an era when companies under public pressure were doing more and now they are doing less, should we expect them to always do less once rules are institutionalized? And this might be an organizational theory question, there might be reasons why, but I, I've never heard a, an answer to that question. And I'd love if you could maybe explain your thinking here and Then and then respond to that quandary.
- [38:38]
E
It's a great question. And a sociolegal theorist who recently passed many years too early, named Lauren Adelman, who was a sociologist and a law professor at Berkeley, did a lot of work during her time on what she called legal endogeneity, which was the foundation of bi. The theory I applied in my book on privacy practices inside companies. And I, I'm going to, I'm going to refocus the question a little bit because it's not a matter of always doing the least compliance culture or endogenous in, in legal endogeneity culture inside a company is not always about doing the least. It's about how companies translate external or exogenous legal requirements into internal practices. And this sits in a classic piece, a classic form of socio legal scholarship. There's law on the books, there's law on the ground, and then there's something in between. This is called the gap gap studies and it's very obviously.
- [39:42]
D
Yeah.
- [39:42]
E
Very technical. Right. So who, who is in that gap? Right. The, the regulated entities are in that gap. And it is the job of in house lawyers, in house compliance professionals. And given the work that Laura, that Lauren did in about human resources, because she did work about compliance, about discrimination in hiring and firing. So she did a lot of work with HR professionals. But there are people whose jobs are inside the company to translate what the laws are and then it trickles down. But it isn't just, it isn't just the HR professional or the lawyer. One of the things that I tried to argue in my book about privacy compliance inside tech.
- [40:27]
D
Can we say the name? And when it came out, Industry Unbound, the Inside story of Privacy Data and corporate Power. And it came out in 2021. Sorry, Plug, but okay.
- [40:39]
C
For our listeners who won't be able to see the, the zoom feed here, I think I see a copy on your bookshelf.
- [40:45]
E
Yeah, it is, it's right back there.
- [40:46]
C
It's got a lovely light blue cover. Yeah.
- [40:49]
E
And Tech Policy Press was kind enough to write a little piece about it and we did an interview anyway. So in addition to it to these people, these entry points, having a lot of power determined to determine like what these structures look like or what implementation of law on the books looks like. You have a lot of other people, right. So you have individuals that translate law into memos, into policies and into structures. And then you have, especially in tech companies, you have project managers who take that information and then help plan how products are being developed. And then you have Engineers and programmers who have to build products and code things based on what, based on some interpretation of what they have to do. And at each point it's like a game of telephone, right? The message degrades and, or, and I say degrade, I don't mean that in a normative sense or a connotative sense. It just literally changes over time because it's being interpreted by people who have different skill sets, who use different languages of communication and languages of power. Meaning the HR rep or the, the in house lawyer doesn't speak the same language or translate the same words as the coder does. Right. So it's not, and what that means to me is that it's really not just a matter of doing the least. You can have that structure and you could conceivably, you know, do something that's not the least. But what that structure does is, or that process, what that process does is it inevitably relies on the way each of those people conceptualize their role in the company and how the organizational structures in which they sit either amplify or minimize that conceptualization and the effects of it. So for example, a product manager or project manager, product manager will have supervision over lots of different teams. They will do lots of different things. And if they conceptualize their role as a, you know, someone who is just getting the job done versus someone who is trying to integrate all of these views, user views, design views, management views, into a more comprehensive product, you're going to get teams that are tasked different with different jobs. Now think about a coder. If the coder conceptualizes their job as simply I am here to solve really cool engineering problems, I don't care about anything else. Leave me to my task. They're not going to ignore or they're not going to be present for other concerns that the product raises. So we, and there have been other legal socio, legal scholars who have written about this like concept of allies already. Like you need allies for a substantive goal inside the company in order to achieve that goal. So if you have allies already, if you have people inside the company who conceptualize their role in certain ways and then you could do more within those structures. And it's not just the in house lawyers like Lauren Adelman in her, several of her, in several of her projects identified how different in house lawyers conceptualize their role differently. They, some of them think of themselves as guardians, some of them think of themselves as business people who are trying to just advance the business of the company. Some of the guardian role thinks about themselves very differently and then they do things differently as a result of how they conceptualize their work. So that's going to happen at any company. But the nefarious and the mischief happens when companies use their structures or use their policies to manipulate people into adopting certain orientations that advance what the company wants as opposed to what those individuals may think is the best. So I would. I think we need to. It's one thing. It's very easy to say, well, any company that's trying to seek profit is going to do the absolute least they can in order to make the. In order to comply with the law and in order to. But to, you know, make the most profit. Compliance there is, you know, a dirty word, but even in the context of a company with people who want to do more and with structures that allow those people to do more. My concern and our concern with a compliance function for trust and safety is its translation of how the mechanism of power, the mechanism of power goes from a holistic, cross organizational, cross department approach, meaning that you have people who are thinking about the rules, you have people who are thinking about better design, you have people who are thinking about lots of all the other elements of making. Creating healthy informational environments. And then you turn it into a thing that a single department is doing, whether that is simply pushing papers or that is just something that a single department is doing, that they consult with other people. One is substantive, is qualitatively a better fit for trust and safety. Pushing papers may be a qualitatively better fit for something else. Right? Where you. All you need to do is just check a box. But it's not a good fit for what we recognize as making healthy safety environments, which I thank Danielle for that phrase, which takes a more holistic structural approach.
- [46:22]
D
And that's what Vidjigati like we saw right at Twitter. You know, I wish I could come up with that concept. Right? You know, sweet. But like Jack Dorsey comes aboard 2019, or, you know, sorry, 2015 is like, we want healthy speech environments. And he empowered his gc Vijay to do that. And so it was a cross, not just trust and safety. It was products they had. What they did was have this very holistic approach because there was C suite commitment post 2016. Right. And so, but that gets. And this is what Ari and I were kind of relying on Ari's work and Adelman's work on legal endogeneity, right. To show that, like, once we're in this, like, stripped down everything we saw the firing of trust and safety employees hollowing out or a Destruction at X, but really like huge firings right across all these platforms. And then of course, they. The rules that exist are in the eu. And so this pivot is not one of these capacious folks who are like, let's think system wise about healthy speech environments, but where you see this compliance culture that Adelman and Ari explored in their work, right? How that translation becomes much more mechanical. Am I right? Tell me, how am I doing? Or you know, like, you know, like that shift with the destruction that we wrote about in our piece, like from 2021 on. Right? Especially right after X, but then even more recently at Facebook, YouTube and others, like Twitch was doing this stuff and then it wasn't, right. We were on the Trust and Safety Council and then we were hired. Like there's like. Right, so that, that's the compliance kind of mindset and that becomes qualitatively different, right? Ari, that is, that's the shift. And that relies on Ari's and Adelman's work on legal endogeneity that I think we've seen play out.
- [48:21]
E
And I think that's why it's wrong or insufficient at least to talk about trust and safety or the end of trust and safety or whatever, as if these firings are the triggers, right? These things are not. This is not new. This is part of a long standing process. And even if those firings didn't occur, the compliance shift to. Of trust, the shift of trust and safety to a compliance single departmental function as opposed to a holistic function, it also would have spelled. Would have been problematic, which is why the economic explanation is insufficient.
- [48:55]
C
I wrote a response, of course, to your piece in Tech Policy Press. One of the things I said in my response was that the piece was sort of slow to incorporate politics. That alongside and in parallel to all of the organizational, legal and economic trends you discussed, there were, of course, many storylines of political drama. And as part of that, you know, was that there was political pressure from civil society, from citizens, from politicians, from government. And those have also, in my understanding, in my view, played an important role in driving the evolution of trust and safety. And so I wonder, do you. Do you disagree, or is there a reason, maybe, that you shifted politics to. The background in this piece is how do you understand the role of politics? That's one. And I know that I've asked you to be quick, so. And that's a tough question, but if you could concisely maybe just give us a flavor of response, I do wonder if my own take on this comes across as too grim. Or pessimistic. Jeff, your cover letter in the Integrity Institute annual report I thought was. Was struck a pragmatic tone about the types of achievements former trust and safety professionals can achieve. Advocating from the outside. And then finally, I want to give you space to talk about responses. And especially, I especially like the. The suggestion that trust and safety workers do things to increase their leverage at companies, including through labor organizing. And so if you could talk a little bit more about that as a sort of political economic response to the trends you've identified. But yeah, maybe we could do Danielle, Ari and Jeff and choose from that. Buffet what you want because I know it's a lot to do in a little bit amount of, short amount of time, but if we could hit some of that in our remaining time together, I'd be really happy.
- [50:43]
D
Okay, so two things in this lightning round that I think we did, I mean by my. I think our view, like gendered violence, gendered abuse is political all the way down. And we are both on the board of the Cyber Civil Rights Initiative and had been pressing and working what from what we, I think, understand as. As politics, right? Like we are engaged in policy at the state, federal level, at CCRI and highlighting what we often ignore, gendered abuse and homophobic abuse is what. That's our project. So I don't think we are ignoring politics. In fact, we are amidst politics in our work on the advocacy side. And the second, in 2016, after the election, I think we wrote about this in the piece. I was invited to meet with Jack Dorsey, the head of. Former head of the New York Times and the head of the J School to spend a day together at Berkeley. Spend a day together talking about the disaster of Russian disinformation and the campaign and how do we create what he wanted us to figure out trustworthy? How do we earn the public trust in an era of distrust and disinformation? And so we spent the whole day talking about how Twitter might do that. It was Jack the Jet Dell, myself, right? And the name, he's amazing, is the name. Two names of the two men beside us is escaping me. But that was all about politics, right? That was about Jack, you know, sitting down for a day to figure out how do we deal with disinformation. Deep, you know, fakery, cheap fakery and farm line influence campaigns and how it's often accomplished by online abuse that's gendered. So that was a day of politics. So do you think we. I thought we were making a case of how politics was in there throughout.
- [52:30]
E
I think Danielle's right. We talk about it. We certainly talk about it. But if giving all credence to your question, if by politics you mean the influence of the current administration and its campaign not only to have its allies buy up communication platforms, but also Elon Musk's very intentional campaign at what used to be called Twitter.
- [52:57]
A
Right.
- [52:57]
E
If you're talking about that, I think it's actually important that we don't attribute, we don't attribute all nefarious behavior to these two people because then we suggest that was an aberration from a long standing trend. Right. Trump is not an aberration from the Republican Party. He is the apotheosis of where they have been going for a really long time. Musk is not an aberration of tech CEOs. He is the apotheosis of where they have been going for decades and decades. And we see it in so many others, from Peter Thiel to Mark Zuckerberg. These toxic men, right, are part of a neo fascist. That's a weird word to say, this neo fascist movement that has finds allies in the, in the current White House. So part of the purpose of our piece was to recognize that this trend toward compliance in trust and safety is not purely an economic story, that it's an organizational story. The extent if we were to argue that this is a result of recent political developments, I think that actually undermine the problem because then, yeah, not only wrong, like I think it would undermine our understanding of the problem because then you could. Oh, once we fix that. Right. Like so, not to say that it wouldn't have been turbocharged, but I don't think it's an apt characterization of what's happening right now.
- [54:24]
C
I do think we're talking past each other a little bit because I agree with everything you both have said. But when you limit it to this piece in the story it tells, I don't think you can tell the story of everything that's happened since January 2025 without also understanding the 2020 election, January 6th, Jim Jordan and the Weaponization Committee. And going back even further to things like Senator John Kyle's audit of anti conservative bias at Facebook. Right. Like these. These narratives about content moderation have been around for a long time and have impacted the industry and the thinking of people like Mark Zuckerberg and Jack Dorsey and led us actually to the things that you just described as sort of the culmination of political trends, not aberration. So I completely agree with you. And yet there's so much story, and maybe there's just not enough page length.
- [55:13]
E
I think that's it. I think that's another two other articles.
- [55:16]
D
That Ari and I have written in this Bain, he wrote a piece called Digital Authoritarianism in UChicago Law Review Online. And we have a piece forthcoming in UChicago legal forum about privacy and democracy's destruction, which does. Those two stories are follow ups to this piece. And it's telling. Yeah, it's. Oh, we got a book to unrelated youth privacy. You want to talk youth privacy? We're there.
- [55:42]
E
So.
- [55:42]
D
But does that make sense? So, like there is a trilogy or plus in this part of the story. So we have other. We could keep talking about politics and we will.
- [55:51]
E
We will cite your review of our piece.
- [55:54]
D
Yes. Right. In a way, for us, it is like an ongoing. There's like two strains of our work together. One is youth privacy, which we haven't talked about and can. And the second right is the relationship between democracy, privacy, speech, online platforms, and content moderation. Of course, as part of that discussion.
- [56:17]
C
Well, we really need to wrap, but I want. Jeff, if you could just say maybe take us out, play us out with a response to Are we all. Is the vibe around trust and safety right now too pessimistic? Should we be looking for reasons for hope?
- [56:31]
A
Yeah, I'm happy to dig into that and to tie it into the politics a little bit. Right. So. So my general take on the politics is this isn't particularly new. Right. Like, it feels new maybe because it's happening in the US but if you look at it from the global lens, this has been happening all along. Right. And you know, India, not too long ago was threatening to arrest all the Facebook employees that were based in India. There's the case of Turkey in the last presidential election. And like, you know, like the deep, deep, sad irony of the fact that, like, you know, Turkey, you know, President Erdogan threatened to turn off all the platforms if they didn't remove his political opponents. Twitter X complied with that. And of course, the sad irony is that Twitter and X had just laid off all the employees that would have helped him navigate that situation to maybe a better outcome because you don't always have to comply with those kinds of things. And so, yeah, the story of what are those people who were laid off and who are caught up in the sad kind of state of the industry downturn right now. And of course, it is sad that we're in this industry downturn. There's a lot of people that, like, I want working on the inside of the platform. Right. Like like, the world is much better off when you have people who care about these things working on the inside of the platform. Right? Like, because the alternative is nobody who cares about these things are working inside the platforms and that's the worst outcome for everybody. So it is sad about the industry downturn, but, like, if you do want a little bit of silver lining, it's a lot of these people are still staying engaged in the space, right? And like, maybe they're not at a big tech company, but now they're, you know, filling out the staff at civil society organizations. Now they're filling out the staff at regulators themselves. Right. And so we're kind of seeing a pipeline being built between having that big tech experience towards regulating the industry itself. And, you know, if you want one little, one little reason to be optimistic about, like, where these things are heading and like, why there's still a little bit of hope in the compliance regime or the compliance era, you know, we did a survey of our members about a year and a half ago around policy and one of our questions was like, would you like to see more regulation of social media? And 84% of our members said, yes, we would like to see more regulation of social media. So regulation of the industry is extremely popular within people who work in it. Right. It's not something that they're scared of or dread. That being said, the type of policy really, really matters.
- [58:48]
C
Thank you so much, all of you. I know you also all have other places to be and thank you so.
- [58:52]
E
Much for having me.
- [58:53]
D
Thank you.
- [59:05]
B
That's it for this episode. I hope you'll send your feedback. You can write to me at Justinette TechPolicy Press. Thanks to Dean Jackson, thanks to my co founder, Brian Jones, and thank you for listening.
- [59:24]
A
Tech policy press.