← The Lawfare Podcast
The Lawfare Podcast
Scaling Laws: The GoLaxy Revelations: China's AI-Driven Influence Operations, with Brett Goldstein, Brett Benson, and Renée DiResta
00:56:28
Loading summary
Transcript56 lines
- [00:00]
A
You know Hannah and I love a good bedrotting session, reality TV snacks nearby and now I've leveled up with my self care game with this Shark Beauty Cryoglow, the number one skincare facial device in the us. Wait, I'm obsessed with it. I've had it for a while actually and it's the only mask that combines high energy LEDs, infrared and under eye cooling. I really need this because nothing wakes me up in the morning. You could do four treatments in one better aging skin clearing, skin sustain, and my favorite the under eye revive with Instachill ColdTech. You put it on and it just feels so good under your eyes. Like I actually feel like I got eight hours of sleep. It's truly like a luxury spa moment while you're literally horizontal. It's perfect for post workouts, Sunday scaries, or when you just want to glow while rotting. To treat yourself to the number one LED beauty mask this holiday season, go to sharkninja.com and use promo code Giggly Squad for 10% off your cryo glow. That's sharkninja.com and use Promo Code Giggly Squad for 10% off your cryoglow Dear Career Ladder, you've had your moment. You're linear and one dimensional. Ambition doesn't just go up anymore. It zigs and zags and squiggles where CEOs, executives, founders. We're advising companies, launching side hustles, taking breaks, defining our next act ambition on our terms. The possibilities are endless. Chief Lead on join us@chief.com.
- [01:39]
B
It'S the Lawfare Podcast. I'm Kevin Frazier, the AI innovation and law Fellow at the University of Texas School of Law and a Senior editor at lawfare. Today we're bringing you something a little different. It's an episode from our new podcast series, Scaling Laws. Scaling Laws is a creation of lawfare and Texas Law. It has a pretty simple aim, but a huge mission. We cover the most important AI and law policy questions that are top of mind for everyone from Sam Altman to senators on the Hill to folks like you. We dive deep into the weeds of new laws, various proposals and what the labs are up to to make sure you're up to date on on the rules and regulations, standards and ideas that are shaping the future of this pivotal technology. If that sounds like something you're going to be interested in and our hunches, it is. You can find Scaling Laws wherever you subscribe to podcasts. You can also follow us on X and BlueSky. Thank you.
- [02:41]
C
When the AI overlords take over. What are you most excited about?
- [02:45]
B
It's not crazy, it's just smart.
- [02:47]
C
And just this year, in the first six months, there have been something like a thousand laws.
- [02:52]
B
Who's actually building the scaffolding around how it's going to work, how everyday folks are going to use it?
- [02:57]
C
AI only works if society lets it work.
- [03:00]
B
There are so many questions have to be figured out and nobody came to my bonus class. Let's enforce the rules of the road.
- [03:09]
C
Welcome to Scaling Laws, a podcast from lawfare and the University of Texas School of Law that explores the intersection of AI law and policy. Alan I'm Alan Rosenstein, Associate professor of Law at the University of Minnesota and Research Director and a Senior Editor at lawfare. Today I'm talking to Brett Goldstein, Special Advisor to the Chancellor on National Security and Strategic Initiatives at Vanderbilt University, Brett Benson, Associate professor of Political Science at Vanderbilt and Renee Diresta, Associate Research professor at Georgetown University's McCourt School of Public Policy and a law firm contributing editor. Earlier this summer, Brett Goldstein and Brett Benson revealed leaked documents from a Chinese company called Golaxy showing sophisticated AI driven influence operations already deployed in Hong Kong and Taiwan. We discuss what this means for the evolution of information warfare, the challenges of measuring effectiveness, and whether the United States is prepared to counter these threats. You can reach us@scalinglawslawfairmedia.org and we hope you enjoy the show. All right, Rene, let's start with some high level context. So you've been studying influence operations many years, from Russian troll farms to sort of everything in between, and more lately focusing on how AI is going to transform this space. So what's changed in the last few years? What's fundamentally different about AI enabled operations versus the old Russian Internet Research Agency model?
- [04:33]
A
Sure. So the Internet Research Agency model relied heavily on human operators and some component of automated. So when people think of the Russian trolls or the Russian bots as they're often referred to, they're actually usually conflating two things. The accounts on Twitter that used automation were the dumb accounts. The other side that they had were the human operators. They actually had a bunch of 20 somethings that were sitting around a factory first in St. Petersburg, they, they moved to Olano also at various times. And what they would do is they would hire these people who they would task with pretending to be in many cases Americans, sometimes other parts of the world. But oftentimes they would try to have them masquerade, operate in the context of being an American Persona. And they had very, very granular senses of what they wanted them to be. So it wasn't just pretend to be an American, it was pretend to be a black woman of a certain age, pretend to be a young right winger, pretend to be a middle aged or an old right winger, pretend to be a liberal woman. So they would have these very granular Personas and then they would have to post in the context, in the first person as a person of that Persona. And what we would see when we looked at those data sets is that occasional they would slip, right? They would get the slang wrong, they would use vernacular, particularly when they were pretending to be African Americans, that just wasn't quite right. And so sometimes to try to get around those slips, they would plagiarize content. And that was another way that we would sometimes find them because they would just explicitly crib from content that was already out there. They would actually, when they were pretending to be young right wingers, they started just grabbing Turning Point USA memes and slapping their own logos on top of it to try to get around the fact that their language wasn't necessarily perfect. So when they were doing this, there were these different ways that we would be able to tell that they were not what they seemed to when you have the capacity to use AI. When large language models started coming out, around 2020, some of us who had researcher access to GPT3, back in 2020, when I was at Stanford Internet Observatory, we started writing about the fact that this was going to really change the nature of influence operations, because what it meant was that you would be able to get around those problems. You wouldn't plagiarize because you'd be able to generate new content and you wouldn't have the problem of the tells, because you'd be able to use models that were going to be able to be fine tuned on content that was of the language and slang from the communities that you were trying to masquerade. As we started writing these risk model papers, Josh Goldstein from my team at Stanford led a really great one that we did with OpenAI, basically arguing that this was really going to fundamentally change how state actors were going to conduct themselves, and particularly that it was going to become much, much harder to find these things because they were going to begin to use them. And over the last two years or so, as large language models became much more readily available to the public, Remember we were writing this in 2020 when these models were still largely gated and only researchers really had access. Once ChatGPT and others began to roll out OpenAI, began to release threat Reports similar to the threat reports that Meta and other social media companies released saying, hey, here are how we are actually seeing in the wild state actors beginning to try to abuse our models. And we began to see those things actually play out in the real world, much as we expected them to happen.
- [08:04]
C
So you mentioned GPT3 and you wrote a great piece in 2020 talking about this future of AI enabled influence operations using sort of GPT3 which was then available and I think you even had GPT3. I mean this was back before, this was back before everyone was using these all the time. So it was still kind of cool to be like, hey, hey, this paragraph was written by GPT3, by an LLM. And you noted that in some ways the capabilities were quite impressive, right? It could mimic your style in a kind of an 8020 way, but it was GPT3, so it's kind of a dumb model. It would kind of get confused and sort of wander off into the wilderness. Obviously in 2025 we're way beyond GPT3, I mean we're two GPTs beyond GPT3 and more seriously, the capabilities have exploded. So before we turn to Golaxy, I just want to ask a follow up question of you. How much have these models improved from the perspective of being able to do influence operations? And relatedly, have we hit kind of diminishing marginal return? So obviously these models will get better and better and better, but you don't need a model that can solve the Riemann hypothesis presumably to do decent personalized influence operations. So I'm just curious, are we sort of 99% of the way to an optimal influence operation model or in five years will GPT 8 or whatever the open source version of it, will that be sort of even worse from a practical influence operations capacity than what we have today, which again is presumably substantially better than what we had in 2020.
- [09:32]
A
So there's a couple things. First, when I wrote the article that you're referencing was something they gave it a kind of sensational title like the supply of disinformation will soon be infinite. I think I wrote this for the Atlantic and I had it as, you know, kind of co author the article with me at the time. I was looking at long form propaganda that the GRU had written in the form of the Inside Syria Media Center. So not just the short form posts of the Internet Research Agency was, which is what I described in response to your first question, but much more long form, the kind of propaganda that I think Goxy will talk about, which is how do you actually also write long form propaganda that becomes the content that actually becomes like training data for future models? Like how do you see narratives on the broader web? How do you make that process of propaganda production essentially? Like, how do you take that way down the cost of production? Basically down to zero. So there's that piece of it so that, you know that, that I think is fairly even. Even back then you had it kind of where it needed to be. It didn't need to be perfect, it just needed to be good enough. But the other things that we're seeing, the audio component is really extraordinary now. The capacity to produce convincing audio so that you can create novel moments that trick people ahead of, for example, an election, right? Like that, that threat model is increasing significantly. How do you create plausible video that can trick people? Right. That capacity over the last even six months is just getting better and better and better. So I think we're going to talk quite a lot about text and Persona based manipulation, which is something that I personally think is, is significantly more impactful in the long run because it does change how people interact with other people. In terms of when you're trying to shape public opinion, you can do that through persuasion or distraction. And when you're doing it through distraction, what we see a lot from China is this model of flooding the zone, just trying to get as much content out there as possible versus what you see from Russia is much more of this persuasion and engagement model of creating plausible Personas and talking to you. When you're creating plausible Personas and talking to somebody, another way that people would detect them was by looking at the profile picture and realizing, hey, this is a profile picture that's stolen. This is somebody else's Instagram photo. Now you have generative AI that generates pictures that are indistinguishable, right? That you, you know, in the early days of generative adversarial networks, the models that would produce faces where there were still a significant number of tells, we would see influence operations, actors use those, but they were still easily detect. The kind of stuff that you're getting through diffusion models now is significantly better. And you can generate the same face in a whole lot of different poses. You can create an entire like backstopped identity with a whole lot of different poses of people with different, you know, different faces, different family members in different places on vacation with their fake kids. You know, you can create an entire identity now in a way that you couldn't even two years ago. So it's not just the text based models it's actually the combination of the ability to create a rather immersive unreality, I think, around some of these Personas that's really gotten significantly more engaging. And then the other piece is the responsiveness of how they engage with you. So the chatbot component in which it is actually able to respond back and have a series of interactions with you, which I want to bring in the goalaxy part of this, because that's the other piece that we would see in the sort of the last bit of work that I was doing at SIO before it shut down, which was the responsiveness of the chatbots that you start to see on Twitter today.
- [13:13]
D
My sense, Renee, is we're still in the early days and we've seen this impressive evolution from what you talk about in Russia, where you had some technically early stage techniques which represented where the world was with data science. We talk about things like clusters and commonalities of groups and you try and tailor it that way, or the techniques, the human side techniques, but we're not really talking about the humans today. I think today we're starting to see this architecture which with gen AI and efficient compute, you can do individual targeting. That's a newer domain at scale. Imagine a scenario where you have a very, very efficient and smart AI coupled with information about each individual. And we start talking about open source intelligence, okay, osint. But we also, in the world we're talking about, there's plenty of leaked information. So the early tickles right now, which I think are going to become increasingly dangerous, are if I'm targeting Renee, I'm going to have messaging only for you. I'm going to create a Persona that's designed for Renee. It's going to be the Persona that can sell you the best, that can be the most compelling, and then as resilient as possible. And when everyone has a Persona that's perfectly designed for them and the perfect message is delivered and you start looking at that in enormous computational scale, that's what's coming and that's what's super complicated.
- [15:07]
E
Let me just jump in right on the heels of that very quickly. I think one of the implications of the scalability and the increasing ability of lots of different groups to engage in this type of behavior is the way that has transformed what propaganda is. So initially, even, even going back to the days of like 1950s, 1960s propaganda in China, analog style, through, in my view, like the 2016 elections, with the sort of dumb types of messaging we had on social media, this type of propaganda typically targeted One type of vulnerable subpopulation, the propaganda that we see now and influence operations, doesn't have that type of selection effect. The potential spread of disinformation. Even if a network, a social media network or something else has been infiltrated, typically because of the persuasiveness of the Personas, the impact isn't constrained by the limitations of individuals because propaganda is much less expensive, because it's more persuasive. You don't have that type of selectivity when it comes to spreading disinformation.
- [16:32]
C
Okay, so I think with that context, we should turn to the actual Golaxy situation. So, Brett Goldstein and Brett Benson. So earlier this summer you published a fascinating New York Times piece revealing documents from a Chinese company called goalaxy. And then there was a great piece in September from the Record reporting how, Brett Goldstein, how you received this mysterious link to a 399 page Mandarin document and you stayed up all night feeding chunks of it into an LLM to translate them. So I would just love for you to kind of walk us through this moment. What did you find? And when did you realize that this was something bigger than just some set of kind of leaked, vaguely embarrassing corporate documents?
- [17:20]
D
Damn. So as you mentioned, it started with a link. And as people know, I hate links.
- [17:26]
C
I mean, the real question is, who's going to play you in the movie of all?
- [17:30]
D
Well, hopefully Brett Benson. So I get this link. It comes from someone who I trust, a trusted researcher. And I still hate links. And I get it. I go through all the things that someone like me should do. I make sure the link is safe. I then put it into a virtual environment and I unpack it and. And then I realize that it's a bunch of weird PDFs, and I start going through the PDFs and I'm like, oh, these are in Mandarin. And I'm not really sure what to make of it because my Mandarin skills are none. And I'm certainly not going to call Bread B yet because it's just a bunch of documents. But then I start scrolling through it, and there's two things that catch my eye right off the bat. One are technical schematics. And I'm deep into a system architecture at this point, and I think it's. From what I can take, it looks like it's collecting massive amounts of data and then producing products, and that's. That's compelling. And then I start seeing big grids of people, and that is very compelling. And then I start seeing pictures of congressmen and I'm like, okay, we've got something here. And that was okay. It's super interesting. I know I need to figure this out. And if I were a typical person, I would dump it into ChatGPT and get some quick answers. But there is no way that I'm going to start dumping pages coming from someone who's passed on the location of these documents that look super interesting into someone else's LLM and get fast translations. So I did the hard path. So as you noted, I was up super late. I'm obsessed with the problem and I'm like, I'm going to install a local LLM in a virtual environment and run everything locally on a controlled network because I don't want to share the answers with anyone. And that's what I did. I stayed up, I built this, I got it going and then I started dropping chunks into it. And that's when I realized that there's enormous OSINT and collect going on here. They are U.S. congressmen and, and it looks like a IO messaging type approach. And I'm like, oh shit, this is something that's, that is super interesting, super hot. And I'm like, where are they going with this? There's neural networks involved, all of that. And that's when I pulled in my research partner, Brett B. And knowing he speaks Mandarin, and I'm like, I'm going to turn the group from one to two and we're going to start to figure this out.
- [20:31]
C
So say more about how this system operates both from what those documents reveal and just from what your reporting and analysis also suggests. Kind of walk us through. What is the smart propaganda system?
- [20:47]
D
Yeah, okay, so this is interesting because there's years of information here. It goes back a number of years. And then based on our research that we searched on for a number of months, going from what was on their website and those pieces, we learned that historically they have one been doing enormous osint like every day. They collect millions of attributes from social media platforms of whomever they're targeting. And in some cases it's about Hong Kong, it's about Taiwan. In other cases that are more murky because I can't figure out what they're collecting. It's looking to the west, they're clearly looking to the west, they're looking at the US they're looking at high profile folks, but it's unclear what they're collecting. So they collect all of this data and they put it into a structure and they make it accessible. But then they have a couple different paths that they work on. One is the creation of messaging. What is the right message for a given outcome against a given target? And I see classical sort of neural network approaches there. But then there's a second part of the effort and I found this, say, both interesting and telling, which is getting at resilient Personas. And what do I mean by that? They want to have a Persona, so a social media account which they can launch messaging from, but also not get caught. And when you start to see line after line talking about how they don't want to lose the account and they want to make sure it's hidden and it stays resilient, you really, really start to get at the intent there. So there's multiple years of history architecture. But here's where it gets really interesting. Before we told anyone what we were doing, we went and we started a deep dive on their web presence and they had more information than you might think up on the site. That has changed as of now, which we can talk about later.
- [23:05]
C
You have to advertise your services somehow, right?
- [23:08]
D
Well, yeah, a little less today. So a couple things that we noted was a partnership with Sugon and Deep Seq and that's where we started to understand their evolution here. So there are things we know they were using traditional neural network type architectures with a somewhat typical stack. And for your listeners, we've open sourced these documents so people can look at these diagrams as well. And that was a traditional way that it grew up. But seeing these partnerships, that's when Benson and I were able to start to extrapolate this is a potential path of where they're going in the Gen AI piece. And as Renee talked about, the world kind of changed a few years ago as we introduced Genai and these techniques. And that's where you can see this whole method shifting from more classical AI neural networks to a generative approach.
- [24:17]
C
All right, let me turn to Brett B. I'd love for you to first give us your side of the story, how you were dragged into this and how your initial sense of this kind of evolved as you dug in. And then also talk about. Brett G. Has talked about what sort of goalaxy was doing, but obviously there's a big piece of this is about Goxy's relationship with the Chinese government. So really curious to hear about that, both specifically to Goxi and also how that compares to the relationship between other Chinese companies and the Chinese government. Obviously, given that it's China, the idea of a fully private company is always a sort of complicated set of questions.
- [24:56]
E
Yeah, how was I dragged into it. That's a good way of putting it, because I feel like I was. So Goldstein and I were traveling, and we were working on something else. We had a very busy schedule. We were overseas. And he one day nudged me and said, hey, I have a cache of documents that I want you to look at. And he said, they're in Mandarin. But he really undersold it, and he never let on that he had already looked at them and had some inkling as to what was in them. And then he gave them to me. And, like, the next day, I told him I hadn't looked at them yet. He seemed a little bit agitated. And the next day, late at night, I pulled them up, and I started to read through them. I thought, Good grief. 299 pages, all in Mandarin. Really dense Mandarin. You know, for those who don't read Mandarin, that's like 450 pages in English. Lots of slides of pictures of members of Congress and people in Taiwan and influential people in Hong Kong. And I had no idea what I was getting into. My background is not influence operations. My background is political economy. And I studied these sort of theoretical models of international conflict. And so I started to poke through them, and it got late at night, and I was pretty shocked at what I had seen. But I have to tell you, honestly, I was really suspicious. I thought, you know, I have seen all sorts of things come out of China, and this is probably in the category of something that isn't really worth our time much more. But when we completed our travels and I started looking at them a little bit more, I. I was. I was pretty shocked by what I was reading.
- [26:41]
C
What do you.
- [26:42]
E
There's one document.
- [26:43]
C
Let me just jump in or something. What do you mean you were suspicious? Suspicious that this was puffery suspicion. This was a kind of a. Its own kind of fake influence operation. What was the nature of that suspicion? I'm curious.
- [26:53]
E
Yeah, it was sort of multifaceted. I was very suspicious. I was worried about downloading it because I had no idea what was in it. I thought it was going to potentially blow up my computer. I didn't know if it was potentially some scam, and I was going to be asked to donate millions of dollars to some influential person in China. There are all sorts of things that were going through my head, and, I mean, Goldstein is kind of a suspicious guy. So I was really suspicious of his suspicion and the documents themselves until I started reading them. So I think that's probably the best way to put it. I started off with one of the Documents, which is a strategy document. It's sort of high level and it's kind of difficult to characterize what the documents and the purpose of the document is because on the one hand it looks like they may be sort of designed to market the efforts of goalaxy to potential donors or something like that. But then some of the documents are also not like that at all. We later learned and figured out that it was basically a disgruntled employee who had leaked the information. And so it was like a collection of documents, some of which were presentations to people potentially to sort of market the capability to clients. There were just a lot of stuff there. One of the documents was a strategy document that explained in very, very vivid detail that the purpose of influence operations is to engage in international informational warfare with certain targets like Taiwan, United States. And it was very explicit. And then it explained the model for doing that, which is to use psychological models in social science, which I haven't had the chance to vet. But they use psychological models to collect enormous amounts of data on open source data on users for the purpose of building these profiles that are highly tailored to their preferences and that are also adaptable. Then the document justifies the types of.
- [29:04]
D
Efforts.
- [29:06]
E
By saying the United States is engaged in these types of things as well. And so China's playing catch up in terms of the connection to the Chinese government, the documents. And it was hard to discern really what the connection is. I think I have a better sense of it now, but when we were reading the documents, it was hard to know whether the documents were over claiming their connection. What the documents say is that gold actually serves Chinese national security and national strategy interests. It's clearly set up by the Chinese Academy of Sciences. For those who know, Chinese Academy of Sciences is a state owned national research institute or academy directly under the State Council, which is the key decision making body for the People's Republic of China. The Chinese Academy of Sciences though, oversees a huge network of institutes and universities. Go one of them. But then some of their clients are the intelligence organizations and agencies in the Chinese government. And then the documents talk about the sort of proof of concept, the validity of the technology as used in Hong Kong and also in Taiwan.
- [30:22]
F
Hey folks, I want to tell you about an absolute disaster I recently had in the payroll department. I have this little substack. It now has an employee. And that employee, you know, I have to do withholding for him. I gotta do multiple jurisdictions, unemployment. And I did it all myself because I'm an idiot. And it took months. I Just wish I had known about Gusto. You know, let's be honest about this. Nobody starts a business for the joy of handling their own tax withholdings. And that's where Gusto comes in, to take the stress out of payroll, benefits and HR so that you can focus on why you started your business in the first place. And in my case, that's because I like writing stuff. I like doing, you know, my morning live streams. I don't want to know about DC versus Maryland withholding rules and how to sign up for the kind of thing. Gusto is online payroll and benefits software built for small businesses. It's all in one remote, friendly, and incredibly easy to use, so you can pay, hire, onboard, and support your team from anywhere. Unlimited payroll runs for one monthly price. There are no hidden fees, no surprises. So if I had like a second employee, it would just be like, enter the name, enter the Social Security number, you're done. Get direct access to certified HR experts if you need them to help support you through any tough HR situation. It's the number one payroll software according to G2 for fall 2025, and it's trusted by more than 400,000 small businesses. If only I had known. So try gusto today@gusto.com lawf and get three months free when you run your first payroll. That's three months of free payroll@gusto.com lawfair one more time. Gusto.com lawfair.
- [32:44]
A
Did you know Tide has been upgraded to provide an even better clean in cold water. Tide is specifically designed to fight any stain you throw at it. Even in cold butter. Yep. Chocolate ice cream. Sure thing. Barbecue sauce. Tide's got you covered. You don't need to use warm water. Additionally, Tide pods let you confidently fight tough stains with new coldzyme technology. Just remember, if it's gotta be clean, it's gotta be Tide.
- [33:15]
E
Are you still running your business with one creaky old phone system, missing calls, losing track of messages, and scrambling to keep up with your team? It's time to break up with the past and say hello to Kuo. Quo is the 1 business phone system with 4.7 stars across 3000 reviews on G2. Quo brings all your business phone calls and texts into one app for your team. No more juggling devices or being tied to a landline. Quo's built in AI logs calls, creates summaries, automates follow ups, and can even answer and route calls. So you never miss an opportunity, whether you're a solo oper or leading a growing team. Huo keeps you connected and helps you deliver standout customer experiences. Join over 90,000 businesses using Quo and see why it's the number one business phone system for customer satisfaction. Level up your workflow with quo. Get started free plus get 20% off your first six months at quo.comtech that's quo spelled q u o.com tech and if you have existing numbers with another service, quo will port them over for free. Quo no missed calls, no missed customers.
- [34:26]
A
Dear Career Ladder, you've had your moment. You're linear and one dimensional. Ambition doesn't just go up anymore. It zigs and zags and squiggles where CEOs, executives, founders. We're advising companies, launching side hustles, taking breaks, defining our next act ambition on our terms. The possibilities are endless. Chief Lead on join us@chief.com.
- [35:05]
C
So how effective has Goxy's operations been? So for example, they were involved in the 2024 Taiwan elections. But of course in those elections I think it's the Democratic Progressive Party, which is the it's called Pro Independence Party in Taiwan won. So as I think I guess with all influence operations questions and I'll start with Brett B. But then Renee, I'd love to get your thoughts on this as well. There's always a question of it's bad whenever they happen, but whether they cause additionally actual bad outcomes. It's always sort of its own separ so maybe a similar way of asking this is how good is go Lexi at its job?
- [35:46]
E
So I think this is the $60 million question. We have ongoing research to try to determine this the effectiveness of this and other like organizations. That's something we don't fully understand yet. As Renee put it, one of the objectives is just to flood the space and to create chaos and confusion. And if that's the objective, then we have to figure out how to measure that. If the objective is to sort of change a mass line or a position of a community on something persuasion. If that's the goal, we have to figure out how to measure and test that in Taiwan they claim, or in Hong Kong they claim to have been effective. But these are the documents making that claim. So nobody has objectively analyzed this because it's really hard to analyze it. It's hard to catch a Persona in the wild and then to trace how effective it is in changing opinions and ultimately having a sizable impact on public opinion. So you mentioned that the Democrat Progressive Party won and the targeted the campaign the PRC campaign in Taiwan, as I understand it claimed through the documents as well some of the people I know in Taiwan was not about flooding the space and creating chaos. It was about driving a particular narrative against the Democrat Progressive Party. And yet the DPP won the election. But we don't know what the counterfactual is. We don't know if they would have won by a bigger margin. Taiwan's infrastructure is set up a little bit better than the United States for resilience on these types of things. Most people in Taiwan know that they're being targeted by propaganda. They know that disinformation is an ongoing part of what China does in Taiwan. And so yeah, the question of effectiveness is still an outstanding one. It's one that we're working on, by the way. But it's also, I think it's one that a lot of people should be involved with because there are lots of different ways to get to this. But it's a hard problem.
- [37:38]
C
Let me actually quickly follow up on that question of how you measure effectiveness. And I'm actually kind of curious about the sort of methodology. I mean, this seems like I am married to an econometrician. So many of my dinner table conversations are about how you do causal inference on very, very, very messy real world phenomena. And like, I can't even begin to imagine how you, how one could. I mean, I'm just a law professor, so it's not my job to figure this out. But like, how could you even in theory figure this out, given that these are very small N situations and there's just so much noise even in the best of situations, let alone where you have a secret subterfuge influence operation going on at scale.
- [38:22]
E
So the causal inference problem is the problem, but the other problem is that we don't know if N is really small or if it's big because it's hard to identify the Personas. We have to figure out first of all how to detect Personas so that we can start to measure. We have to measure detection before we can measure the scope and impact of the influence. Then we have to set up. This is one of the things that we're working on, but we have to be able to set up some sort of an experimental type of a situation that's designed to, to get to the causal identification problem, to get to what would the counterfactual be in the circumstance in which a particular target was not targeted. And right now most of the information on this and most of the evidence we have is purely anecdotal. I think that you've hit the nail on the head that the Causal inference problem is enormous.
- [39:20]
D
So I think just pulling back for a second, one of the reasons I'm worried is an experiment we did over a year ago. It's a different use case where what we did was pulled an individual's information from LinkedIn, so very, very limited OSINT, and then we pulled it into an LLM and we created persuasive messaging. And the crux of the idea was to try and get an individual to engage, presenting a paper at a faraway location. And something that like a lot of us will get an email and we'll just send it to Arxiv and be done. And so what we did in the experiment, we pulled LinkedIn, created persuasive messaging that was individually tailored for that person and we generated an email and then we chased it with an AI, generated audio engagement that called the person following it up and we were getting 70 to 80% click through on these tailored messages. And that's like a five minute technical effort at this point. Nothing at the level of sophistication that we should be talking about here. There is a technology in play which is super awesome gen AI which has the ability to customize messaging for individuals that gets them to engage in ways that they typically don't. And that's some new terrain. So that little experiment from over a year ago coupled with this disclosure has me very, very curious about really what we're walking into.
- [41:12]
A
So I think there's a couple things here. First, I agree that the threat model is significantly different. The papers that we wrote at SIO on this also around persuasion found much the same thing. My colleague Josh Goldstein again found similar things with testing persuasive messages. There's a DARPA program called Active Social Engineering Defense Aced that looked essentially at the same thing. Like precision spear phishing. It's called the ability to target high value targets with precision messaging is significantly easier. Now people are often unaware of how much information they are putting out there about themselves. Anybody with an Instagram account is telegraphing where they are, their kids names, everything about them is out there. What you say on Twitter, the photograph that you post, people can intuit where you live based on the type of bricks in your wall sometimes, right? So there's a lot that you can gauge. And if you are a high value target, that kind of dynamic can happen. And one of the things that if you look at the goalaxy documents, and I did run them through normal chatgpt, you do see the sort of folks that are in there, the sort of high value Politicians and who they want to engage with. Interestingly, when we were at sio, doing our work, would write up content on Chinese influence operations, and we would get the Twitter accounts of reporters at Chinese state media would land in our DMs. Right. Hey, read your stuff. If you ever want to talk to us, just send us a dm. Right? I used to work in the ic. I know what that is. And I would have to tell my students, do not engage. But I remember having once we actually wrote a report on the U.S. pentagon's influence operations, and the Chinese Ministry of Foreign affairs was tweeting it, congratulating me on my excellent report. You know, so. So there's the way in which these things play out is sometimes the only.
- [43:13]
C
Thing I can tell you. Renee asked for a certificate suitable for framing from the Chinese government, but it was a.
- [43:19]
A
But here's. Here's what I'll say, though. So Josh and I actually have a paper coming out in Security Studies that looks at the bragging in leaked documents from some of the contractors that perform work and some of the state. And some of the state outlets that run these influence operations. There's one called Fabrica that runs stuff on behalf of the Russian government, sort of, again, one of these sort of semi state entities in which they brag about how only 1% of their accounts are getting found. Right. And they're talking about running influence operations against Ukraine and how they're successfully changing public opinion about Ukraine, how they're making people think that Ukrainian local mayors and things are corrupt, making them think that they're driving Bentleys and going on vacations and owning dachas and stuff like that, which, if you read the media outlets, like that kind of stuff is actually out there in the ether. Is that coming from these accounts? Who knows? Because as Brett Benson notes, that it is very, very hard to trace the origin of a message and say, this came from here and here is how we can trace it through. And this is one of the challenges when we talk about propaganda. It is a very multifaceted dynamic. And even what we saw with the Internet Research Agency, which the stuff that happened from 2015 to 2018 or so was three years almost of uninterrupted effort. There was no social media platform that was looking for it and taking it down. There was no Integrity team anywhere at a platform that was saying, hey, we're looking for these foreign accounts. We know they're here and we're going to try to disrupt them. Which is what is happening now and has been happening since about late 2017 or so. And so what you see there is these accounts managing to amass followings of several hundred thousand followers in some cases where they're speaking in the first person as a messenger who looks like a fellow member of the community. And that's very important too. That's why they're speaking in that first person tone. They had accounts that looked like, you know, niche media, but that's not what hits, right? It's the accounts that look like somebody who's just like you. This is why, like I wrote a book on influencers. The reason influencers are such successful propagandists. And the reason what you see is the Russians evolving their strategy from trying to pretend to be these influential figures to just straight up paying influencers, right? And if you remember, but tenant media is this thing that the DOJ and FBI go and they put out these, they indict them, this, this company. Because what they start doing is just straight up going and paying right wing influencers to say the things that they want said about Ukraine instead of bothering to create these Personas because they know that the social platforms are out there trying to disrupt these accounts. And if they can get the authentic voice to say the thing that they want said, well, that saves them the trouble of having to do all of this work. And that's because building these accounts and actually doing the things that they're claiming in their documents that they can do. The claims that we see in these documents versus the reality of the impact, we have not yet seen that similar degree of success as we had seen in the 2016-2018 timeframe in terms of actual significant impact at that level of actually managing to amass an audience and having these sustained discourses where they become essentially vox populi and they're getting embedded in news articles and amplified by prominent people, and we're really seeing them represented in the, in the discourse. So the promise of what they can do and the technology has improved and the threat model is there, and that's all true. And they're bragging about it in their documents. And you do see, again, some of the narratives that they claim they're seeding out there in the world, but is it, you know, can we make this causal link? And the answer is like, it is very, very, very hard to do that. The only time you can really say this came from that is when you find some hashtag that they actually claim that they got done. And you can say like, okay, this viral moment was tied to that thing we have not yet seen that from a Chinese network that I can recall in all of my years of seeing China try this stuff. And so that's where I'm like, you know, one of the things we have seen from China, though, is in their leaked documents, they will brag about their metrics. We posted this many messages, we did this many things. And they're talking about numbers and engagements, like, we're checking the box, boss, look at this. We've done this many things, but they're not talking about getting feedback back. So I too can run a bot that pushes out 200,000 messages in an hour, but that doesn't mean that anybody's engaging with them. And when we would write our response reports, we would say most of these accounts never got a single response. So they would be out there pushing messages about the Uyghurs, but they would get almost nothing in response. And that's where I think the thing that they recognize, and you see this in the documents. When I read them, what I saw was a recognition that they are significantly behind Russia in terms of their ability to actually generate any kind of meaningful engagement with an audience.
- [48:27]
C
It never ceases to amuse me that at the end of the day, every bureaucracy is exactly the same. People are just all about how many boxes I checked. Because that's easier to justify to your boss. Okay, let's now turn to the US Angle of this and sort of close out with that and what the US can or should do about all of this. Brett G. When you described initially how you came across these documents and what jumped out at you, you mentioned that once you started seeing pictures of US Congressmen, congresspeople, that's. That was your. That was your oh shit moment. So dig into that, Dig into that more. What do we know about gox's operations or capabilities? Even not just in the kind of Chinese purported zone of influence of, you know, Hong Kong and Taiwan, but in sort of the good old US of A.
- [49:18]
D
Yeah, so. So it is in fact limited what we know about the docs. Like, we see we have lots of headshots, we have a system architecture. It's not clear on what they collect. Where I transition the conversation to is when we talk about what this technical architecture can look like. It's not really speculative at this point. And you know, I know in some worlds of technology we're like, yep, in three to five years we're going to be able to build X, Y and Z. In this case, I think about it as things I could potentially build on my laptop. So we have to make certain, I guess assumptions here, which I hate. But you look at goalaxi and you see all these names and faces. Why wouldn't there be enormous OSINT collect on Fuchs? This is no longer a hard exercise and they've done it in other regions. Two, why, why wouldn't it be a collection on more folks? And it's. We have a couple screenshots where Benson and I think this is just a representative, not complete cohort and why would there not be broader collection? And then when I talk about sort of technical ease at this point, like my example of how I started to unpack the documents, I can run an LLM on my laptop. This is not an enormous data center that is required to do this. You can think about and I'm reflecting on Renee's comments before when she talked about high value individuals. I look at them like you can get any value individual now because it's simply a function of compute. So the things we think about in the US right now are what does it mean to have the enormity of this data being collected? And we talk about in smaller scales like TikTok and things like that and all the data and the exhaust that is out there. Two, the ability to use a variety of gen AI tools. You know, some there are protections on but if you go to hugging face you have so many choices that you can do anything you want with. And then the generation of messages. But if tying that all together And I was in Japan last week and we started getting phone calls from reporters asking about the video generation tools that are coming out. And this is all starting to tie together now because we're seeing propaganda that comes out in a variety of modalities. And I think Renee earlier talked about multimodal methodologies. How do we start to base case, discern what is real versus what is not real and then start to be able to. I'm not really into the censorship side of it, but instead be able to effectively, and I'd say at wire speed, know that something is created by an AI and then be able to more broadly get at the networks. And that's something that Benson and I are working on because this is a problem that is speeding toward us and has become even more real over the past couple of weeks.
- [52:52]
C
So. So what do we do about this? I'm going to start with Renee, because I know you spent much of the last few years thinking about this. Correct me if I'm wrong, but it does not seem that the US infrastructure, whether governmental or a private platform is well Equipped at this moment in time to deal with this problem. So with that provocative statement, Rene, give me your thoughts.
- [53:21]
A
Well, I have thoughts on that.
- [53:22]
C
Yeah, I bet you do.
- [53:25]
A
Well, SIO doesn't exist anymore. Why is that? No, the fact is.
- [53:34]
C
The way that.
- [53:35]
A
We did a lot of our work looking at these things was we would see stuff at Stanford and we would reach out to platforms and we would say, hey, we see these things. Do you see these things? And we would say, and the process of attribution saying, hey, this is actually a Chinese network, because. Or Iranian or Russian or whatever it was, when we would try to gauge the attribution was done jointly and it was done because we had some piece of the puzzle. They had some piece of the puzzle, and that was done quite collaboratively. And then sometimes even the briefing to the government, depending on what it was, sometimes with the Russian stuff, when it was like Wagner Group or something, that would be done jointly too, because there was a multi stakeholder effort at the time and that was reframed as some sort of weird collusion, right? Some sort of weird cabal. And those ties were broken apart and that doesn't exist anymore. And then you saw the platforms, some of them really roll back the teams that they had in place to look for this stuff. And I actually went looking when I was ahead of our chat today to see what the most recent reports were from OpenAI, which put out their most recent threat report. They put out these quarterly threat reports that are out there for the public. Anybody can see them. And again, the work we did at SIO was also released to the public. Anybody could see it. OpenAI released theirs on October 7th. Meta hasn't put one out since May, so I'm waiting to see what their next one looks like. But, but these reports, Twitter hasn't put one out in ages. I mean, God, I don't think since Elon bought the company, they've released one, but they used to put out these reports just letting people know what they had taken down. You could go and you could look at artifacts, you know, tweets that, that were representative of what the network had put out. You could actually get complete data sets that were, you know, where the usernames were redacted, but could go and look at the content and things have really changed. So even as the threat model has increased, the capacity for detection has decreased. And with the last sort of work that we were looking at, trying to gauge the automated accounts specifically, you could sometimes see LLMs slip up when a user would engage with them. Or when they would put something out, they would be responder bots. Right. A lot of the times, the bots would be tasked with replying to accounts, and something would trigger a violation where you would see what we called slips, and they would come out with something. They would spit out an error message, and the error message would say something to the effect of, like, as a large language model, I cannot. Now, that particular one became a meme. Right. People would start using it, actually, so that one became relatively useless as a detection tool. But you could see these other ones, and oftentimes they would be in the realm of cryptobots. But as the detection got harder, the capacity decreased, and then the state capacity in the US Government decreased also. Right. You started to see some of the entities in the US Government that were tasked with this being being stood down. So it's unfortunately, you know, what. What do we do? Well, I mean, we rely on new entities being stood up. Maybe we rely on private sector detection capacity. We hope that people are paying that $40,000 a month for the Twitter API that. That it costs now when it used to be free. I don't know. I mean, I candidly don't think we're in a good place, really. I think that as the capacity and the threat has increased, the ability to detect has actually decreased. And I think that that's actually quite bad. And this is not a partisan issue. This is a national security issue. And we should really be rethinking what we've done over the last. Over the last ten months or so.
- [57:13]
E
Yeah. One other thing that I'll mention, as Brett Goldstein and I have looked at this, and especially in the aftermath of the New York Times article, I guess I've been a little bit surprised that the most common response is, well, understandably, one response is a little bit of paranoia. Am I being targeted? The second one response is really understandable, and that is, does that mean our democratic institutions are vulnerable? How effective can foreign influence operations be in influencing the population, undermining these sort of common civic beliefs, undermining our elections? And then the conversation stops there. There are some national security problems that we have been thinking about that I think are just as problematic and concerning one of those is that, historically speaking, because the US Defense posture is one of deterrence, the United States military typically treats information as a supporting function for kinetic and diplomatic operations. But our adversaries are treating information as a battle space itself. What is the implication of this? The implication is that before a crisis even begins, there is a battle taking place over Information itself, which distorts the ability of the United States to effectively deter. And then this plays out in lots of different ways. I mean, this is sort of a common problem for gray zone conflict. But this type of gray zone conflict is much more problematic because it's ongoing and never stops. Which means that it's a little bit different than some gray zone kinetic operation in the Taiwan Strait that could, you know, something like a fishing boat seizure or something like that, that where it happens and then you find a solution to it. This is ongoing, non stop and so it creates some strategic exhaustion. So there's a strategic level problem when it comes to national security. But then there are also some operational challenges about dealing with influence operation in the gray zone. And I don't think that, I don't think that we're quite ready to think about. Well, I think we're thinking about these problems, but we're not ready to address them. And that is that influence operations are potentially. Well, not potentially. I mean this is the challenge. Influence operations like the ones that we have been discussing are inevitably going to be partnered with cyber attacks, economic coercion, military attacks. And so these are going to be combined domains of coercion. And we're not quite ready to deal with that just because our defense posture has been one that's reactive as opposed to sort of forward looking. The last thing that I'm concerned about, and this is one where democracies could have an edge in the long term. Right now we're vulnerable and that is that influence operations can create like wedges between democratic allies just by influencing the beliefs so that all the actors, the allies, priors are different when they, you know, they're looking at different data, they're looking at different, they've been exposed to different types of influence operations. This is something that we need to start thinking about and engaging with our allies about much more intensely than we are now.
- [60:41]
D
I think we're at this interesting inflection point when we think back. Look, technology is awesome. Go back to the dot com boom. We did amazing things and it's been awesome. But take the analogy of cybersecurity. We started the dot com boom and we have been doing patches and responses in zero days for decades now. And we are constantly reacting to that. Look, ripped from today's headlines is the F5 issue. Like this has been decades of this now. So we now in this space we see a new issue. And I think the question for government, academia, the private sector is are we going to do what we did with cybersecurity and be constantly reacting to the next threat? Or is there a way that these different groups can come together and let the technology advance at the speed which is amazing, but actually get ahead of the threat in a different way than we've done in cybersecurity?
- [61:48]
C
I think that's a good place to wrap up. And I suspect, sadly, with the the midterms around the corner, this is not the last time we're going to have an opportunity to talk about AI generated misinformation, whether foreign or domestic. So we'll continue this conversation. But for now, Renee Diresta Brett Goldstein Brett Benson thanks so much for coming on the show.
- [62:07]
B
Scaling Laws is a joint production of Lawfare and the University of Texas School of Law. You can get an ad free version of this and other Lawfare podcasts by becoming a material subscriber at our website, lawfairmedia.org support. You'll also get access to special events and other content available only to our supporters. Please rate and review us wherever you get your podcasts. Check out our written work@lawfairmedia.org you can also follow us on X and Blue Sky. This podcast was edited by Noam Osband of Goat Rodeo. Our music is from Alibi. As always, thanks for listening.
- [62:46]
F
Listening.
- [62:51]
A
Dear Career Ladder, you've had your moment. You're linear and one dimensional. Ambition doesn't just go up anymore. It zigs and zags and squiggles. We're CEOs, executives, founders. We're advising companies, launching side hustles, taking breaks, defining our next act ambition on our terms. The possibilities are endless. Chief Lead on join us@chief.com.