← Just Now Possible
Just Now Possible
Beyond Black Box Scores: How Musubi Trains Custom AI for Trust and Safety Teams
01:12:40
Loading summary
Transcript96 lines
- [00:04]
A
Welcome to Just Now Possible with Teresa Torres.
- [00:09]
B
Hi everyone, I'm Nikki. I'm a data scientist at Misubi and really by that I mean I'm data science and background. But at Misubi all of our roles are a little bit ill defined or we play a lot of different hats. So I also am helping a little bit more on the product side of things is we reinvent some of our products, trying to navigate there and prototype new things.
- [00:35]
C
Hi there, I'm Brian McCaffrey, I'm a software engineer at the team at Misubi. Now I like Nikki, wear a lot of hats. Typically I'm more focused in the back end but I also do some front end prototyping and really have been focusing on helping our customers find the right balance of latency, accuracy and cost when it comes to serving our custom trained models.
- [00:59]
D
I'm Dan Means, I'm a machine learning engineer here at Misupi. I primarily do most of the verbs that are related to machine learning models and LLMs, training them, evaluating them, deploying them, automating and creating pipelines around them. Before this I worked at Cloudflare as a part of their machine learning team for bot management. So basically for a while I've been an Internet janitor.
- [01:23]
A
I love it, I love this framing of you do most of the verbs. Awesome. Really excited. I love it when our episodes all my product management people hate it when I say this, but I really am learning that I love episodes with engineers and data scientists because it's really where most of the fun and AI products are.
- [01:39]
C
It's actually pretty interesting. So at Misubi we don't have any product people like defined product roles and everyone has to wear a product hat. So everyone at the company talks to clients and actually like talks with our end users in some capacity when we're developing features. And it's really cool. I've always felt personally that like engineers who wear product hats and anyone who wears product hats like make really good products just because they have the end user in mind. So it's cool to see a company that's no like inherently product people on paper but everyone is a product person.
- [02:16]
A
This is what I love, right. It's not that I'm anti product people, it's that I think everybody needs to be a product person. And I also think humans are builders by nature, like creators by nature. And that the more product people that don't have these build skills like lean into these build skills like it's better for everybody. So like I love it both Directions like engineers that become more product minded and product people that become a little more technical. Okay, tell me a little bit about what does Moosubi do?
- [02:45]
D
We are a trust and safety toolkit for AI forward platforms. So trust and safety teams are the folks that help keep your platform or website safe from some of the harms that you can imagine could happen on a platform. So you know anything from spam to fraud to any harmful content or things like that, or maybe bad actors on your site. So there's a couple concrete examples so you can imagine. If you are a marketplace, you probably don't want fake reviews or fraudulent transactions or impersonator accounts. If you are any site with user generated content that could be a social media site or even like a new site with comments, things you probably don't want are like maybe extremely harmful content on your platform, you probably don't want spammers, et cetera. You probably want a healthy conversation that's happening. Or if nowadays you're an AI company and you offer some type of like inference endpoint where users can ask for something and it produces some type of output, you probably want to have some guardrails around that. And what you can and cannot produce, for example, like copyrighted content, maybe extremely violent content, et cetera, these are all things you want to protect. And so that's what trust and safety teams do. And so what we try and do as MISUPI to help empower these teams are basically a couple things. Like, one, we want to give them visibility into what's happening on their platform and what the activity is and how their moderation activity is working. But the second thing we want to do is basically help them take action. And so we essentially help them either deploy AI moderation teams or trust and safety teams that actually take action on their behalf as well as give them the toolkits that are needed if they want to deploy a model themselves and go into production. What are the things that they will require in order to help them have an AI forward trust and safety team. So that's a bit about Misubi.
- [04:29]
A
Yeah. And tell me a little bit about the sort of stage of company Misubi. Like are you an AI native company? How long have you been around?
- [04:37]
D
Yeah, maybe two, three years. And I think we're, I would say we're AI native in the sense that I think from the start, like our first product was an AI moderator. Basically. You can imagine these trust and safety teams have groups of folks who are reviewing this content and the idea was like, what if we can empower them by having an AI agent on their team that was doing the same task. So I made it from day one, and we've been building a platform based on that Start.
- [05:01]
A
You know what I like about this problem space is that this is a part of the Internet that I think a lot of people aren't aware of. And that historically, especially like the metas of the world, like these large sites where we see some of the worst of human behavior, these companies are like hiring humans in Africa to look at hours and hours of just atrocious content. And there's a lot of research coming out. This is bad for humans. To me, this feels like the perfect job for AI. Like, AI is not going to have mental instability because they spent eight hours a day looking at the worst of humanity. Obviously, we probably need humans in the loop somewhere because we have to train the AI, but if we can dramatically reduce how much the humans have to be involved, I think that's probably a net positive for everybody. And I think the other thing I like about this space is this is a genuinely hard problem. It's one of those problems where the targets are always moving. It's not like you just define this is what spam looks like, or this is what malicious content looks like and you're done. It's just constantly evolving. And I think that's also where maybe some model intelligence can help as well.
- [06:07]
B
It's so true. I didn't actually realize what this industry was before getting this job, but I've since watched some documentaries around moderators. And yeah, people get traumatized because it's just like, at the very best, it's very monotonous work because you're just looking at content, like thousands of images per day. But at worst is very traumatizing. And the way that the system works to have that human in the loop is we automatically approve things that are obviously safe. We automatically ban things that are obviously banned, like they should be banned. And then anything in the middle can get escalated to a person for review. So that's like a nice balance where you're reducing that monotony, you're reducing the very bad stuff that people are exposed to. But then you leave those gray areas and the nuanced cases to the human team. Yeah, and that's. Yeah. And then the same one for adapting too. One of our systems learns based off of human decisions for the human moderation team. Something that's really nice there. And I think what trips a lot of people up is you might create some sort of machine Learning algorithm. And it does great at first, and then the spammers figure it out and they adapt and then it becomes this game of whack a mole where you have to figure out new features, engineer. And by learning what the moderators are picking up on, we're able to adapt automatically as spammers try and change their tactics.
- [07:35]
C
Our co founder, Tom, was the CTO at Grindr. And just as our head of trust and safety, Alice was also at Grindr and beyond the human cost is a lot of these businesses look at trust and safety teams just as a cost center. This is something I have to do to protect, to check a box. And Alice has really been trying to coach us and I think where we want to take our product is leveling teams up so that they're no longer just struggling to breathe, they get above water and they start to thrive. And businesses can actually look at trust and safety safety teams as thought partners and really just valuable contributions to the entire business and not just people who tell us we can't do things or ban people from our platform.
- [08:16]
A
Yeah, I love the way you guys are framing this. Nikki, you talked about your software can get rid of the worst so humans never have to see it and just the mental health benefit of that. But it could also auto approve the best, like what's clearly fine. And then I think what's hard about content moderation, there's a lot of gray, there's a lot in the middle that probably needs human judgment. And so in some ways, everybody's afraid that AI is going to take their jobs. My first shallow reaction to this, like, maybe these are jobs AI should take because no human should have to look at thousands of atrocious photos all day long. But it's probably not even going to take all of those jobs because that gray area is so big and there's so much nuance and it's so context dependent. And I think maybe today we aren't even making the best decisions in that gray area because we spend so much time on those outliers that are so just where the yuck factor is really high.
- [09:12]
B
Yeah, that's exactly it. And we hear that from our customers a lot, is they're spending so much time in this reactive mode because they're just trying to stay up with their moderation cues and it's like really just stop the bleeding. And they really want to shift their work to being more proactive. If we can automate a lot of those decisions, spend more time on the nuance, you can spend more time figuring out the exact boundaries on your platform that you want to enforce. And then Dan mentioned this visibility layer. You can actually start to try to look for cases where it's not getting through your moderation system. Like, you're just missing it on the platform. It's not even going into your queue in the first place. So it kind of shifts the work into being more, I don't say like, knowledge work, but it's more like specialized, proactive work to make sure that the content that's on your platform isn't just bad or, like, you're not just getting rid of the bad stuff, but you're actually fostering what you want. Like, you're making sure that it's actually good and what you want.
- [10:08]
D
And on the topic, too, Nikki, of, like, teams keeping up with kind of the onslaught of things they have to moderate, I think there's actually, like, a simple, like, economic reason to use AI too. And that's, like, nowadays it might actually be cheaper to generate this kind of content or spam behavior than it is to review it. Right. So we have this, like, asymmetry between the adversarial nature of this thing. And so it's almost like teams aren't choosing whether to use AI. It's like we are using AI. It's kind of one of the only ways to keep up. It's more of a question of, like, how.
- [10:38]
A
This is a really good point that I hadn't really thought of. We're seeing this, like, across the world in a lot of domains right now. Whether we're talking about, like, the Ukraine war's impact on, like, drones being shot down with Patriot missiles, which clearly isn't sustainable, or like, the one that I've been thinking about a lot because of the last couple of weeks. We're recording this on May 28th, and I'm not going to get the name right. I'm going to call it the Tanstack hack. It's like a package hacking, like, worm that is spreading across the Internet. And it's like one of those things where, like, we've always had hacks like this, but the volume and, like, the number of people who now know how to do these things, because it's not that hard. You just have to have a little bit of knowledge. And now LLMs can just tell you how to do it, but to defend against it is. There's asymmetry there. And I think we're. We saw this also with all of Anthropic's narratives around Mythos and can we release Mythos to the world. And so I think we're like this has always been true with technology, but I think like in this moment in time we're really starting to see this imbalance and then how do we use the same technology to bring it back into balance? And I didn't really think about that From a content moderation. I feel like it's always been out of balance, but maybe we're at a point where we can bring it back into balance. And how great would that be for the Internet? Okay. I think we have a clear handle on the problem. Space you mentioned the company is two or three years old. Tell me like what is your each of your tenure at the company? Were you around in the beginning?
- [12:09]
D
Yeah, I came on as the second engineer essentially at the company. So it was like a couple co founders and we had an initial founding engineer. And so basically where I started was we were launching with our first customer. So we had this prototype and we essentially wanted to see whether or not this product would actually work live in production on their platform. So that was like my initial point.
- [12:29]
B
Yeah. And then I came in probably six months after you, Dan. So I've been. It was supposed to be about a year and a half and my entry point, things went well with that customer. So like spoiler alert there. And then we had to work on scaling out traditional customers and taking this type of this adaptive machine learning pipeline that we talked about and making sure that could scale to different customer types so they have different data and they have different problems on the platform.
- [12:59]
C
Yeah, I've been at NASUBI for about six months now, which is wild to think about how much we've accomplished and grown in that time and just speaks to the age of AI that we're in. And yeah, it was really brought on to help. There's a client, a couple clients that wanted really low latency inference and our current production facing model just, it has pretty good SLAs in terms of time. We're talking like single digit seconds. But this, you know, some clients wanted sub second and so that, that's where I was really excited to help out with and what we've been building since before I joined and since I've joined.
- [13:33]
A
Yeah, amazing. Okay. I think we're going to get to go through the whole narrative starting with Dan and that first prototype all the way through to this hard problem that Brian has brought on to tackle. So Dan, let's go back to when you first joined and you framed it as you were just starting to put a prototype in front of the first customer. I can see immediately how AI can help with this problem space, but it also feels enormous. And so I'm curious about how did the founders find their first starting point? What did that first prototype look like? I'm assuming you didn't go to a customer and say, we'll solve all your content moderation needs. What was that first slice?
- [14:13]
D
Yeah, I think the term slice is a really good way to put it where I came in. I was fortunate that the founders had set up an awesome foundation where essentially our founder Tom, as Brian mentioned, experienced the harms of trust and safety and how difficult was to manage firsthand. And our other co founder, Phil, basically had the idea of what if we use ML and AI algorithms to essentially address this problem? And so what we're initially doing was training on just like a sample of data. And you can imagine it almost starts off what seems like a school project where you're like, look, it works, but the question is how you actually take that and make it go live on a platform and actually have it continue to work day in, day out. Right. So at the initial starting points, like, we had the idea this works in theory, we have the proofs that it works and the question is, like, how we actually roll it out. And so like a couple things that I remember us doing or maybe focusing on, as you said, slices. So essentially like, what are areas that we can have high impact but also not completely unleash an AI across their entire platform all at once. And so what we did is focus on certain tangible aspects, like you can imagine, like spam is something that an AI would be really good at catching, and maybe certain things like illegal activity on a certain social network that we're working on, There could be some drug dealing and things like that that we could catch. And we focus on some initial problem sets. And then over time, once we gain confidence and deploy data into production, then we stepped on to widen our purview.
- [15:42]
A
Yeah, okay, because you mentioned spam, you know what immediately came to mind? And I'm going to date myself with this comment, but I immediately thought of the WordPress plugin, a kismet. So if for people that were on the Internet, like In the early 2000s, comment, spam was a nightmare, right? And then this plugin comes out. It did a reasonably good job and suddenly this like, hard problem was, let's say, 80% solved, I'm pretty sure that wasn't. There clearly were no LLMs at that time. You've mentioned, Dan, your background's in machine Learning. I'm really curious, like on day one, like with this prototype, what was the mix of just more traditional machine learning versus where did LLMs play a role? Was something meaningful unlocked with the introduction of LLMs in this environment? Talk a little bit about that.
- [16:33]
D
Totally. I think that's a really good point. I think where we started with spam, traditional machine learning we found performed really well, particularly for the kind of data that we are getting. So particularly what we were getting is basically what you would call tabular data. So you can imagine like a lot of different features or columns that describe like maybe a user profile. And basically that worked really well with categorizing content. But then the question is like, what kind of intelligence did we need as a platform to build a service to a customer production? And some of the things were, for example, when you make a decision as a moderator in general, you have to say the reason why you made the decision. Right. So often you have to categorize the reason. And as moderation teams get more sophisticated, some of those reasons become pretty specific. So it's not just like someone is being sexually explicit. You can imagine a bunch of subcategories in that area. And so it becomes really a lot more difficult for maybe a traditional machine learning model to have that fine grained accuracy there's Right. And so that's where we started pairing like LLMs to say, once we've made this decision, review the content, just hand off to an LLM to say, what specific sub reason would this be extricated to?
- [17:40]
B
Another way that we're doing both and kind of integrating them is more on the feature engineering side. So LLMs are just fantastic. You can tell it whatever you want to extract from an image or through say like a message history, for example, like maybe you could say let me know if there's any nudity in this image. Or like there are other, like there's Python libraries that can do that for you too. But the nice thing about an LLM is you can make it very specific to the customer that you're serving and like their nuanced policies of what's allowed and not allowed. So that's one thing that we're exploring too is using LLMs more as a feature engineering step and then feeding those into the more like traditional ensemble models to pick that up as signal.
- [18:24]
A
Yeah, okay, Dan. The way you framed this of the moderator has to give a reason. What went through my head was, oh, this is just like qualitative analysis or even error analysis looking at traces. We're really Just looking at content and we have to determine. I can see how a machine learning classifier could, like good, bad, like a, at a. Obviously it can be more sophisticated than that, but I can categorize things. But then you still need this almost the why behind it. And it's funny, this pattern emerges across so many domains. We have a lot of data we're looking at how do we use an LLM to not have to have a human look at piece after piece after piece. So I find that fascinating. One thing that's fun about this podcast is these little patterns almost emerge across all the different types of products. And it's. Oh, I can see how just in a couple years we're already starting to identify these, like, really core building blocks of like, how to build good AI driven products. Okay. If I understood correctly, it sounds like you're doing a couple of things you do. You're still relying on traditional machine learning to kind of classify things, bucket things. There is some large language model reasoning intelligence around maybe even more qualitative labels, more even getting really granular with those labels. Tell me a little bit about. Maybe we can even stay at the beginning of this prototype phase and we'll obviously move towards where you are today as well. But for that first customer, I'm assuming they had a moderation team. Was the goal of your product to mimic what the moderation team was doing, or was it to make what they were doing easier?
- [20:17]
D
I think it's definitely the latter. Right. I think it's definitely to empower moderation teams. And it's interesting you bring up mimicking because one of the first ways of determining whether or not we're doing our job well was to try and find some benchmark of comparison. Right. And as you mentioned, it's entirely subjective. There's no objective truth of whether or not this should have been taken out, et cetera. And so I almost feel like what we're doing is just creating benchmarks, just as LLM companies do. There's no such thing as intelligence. Right. Where they're just creating benchmarks that have some measure.
- [20:46]
B
Right.
- [20:47]
D
And so in this case, the benchmark we created was just comparing ourselves to moderation teams by basically taking actions and withholding some percentage of those actions in order to compare and see, like, how well they would do against those teams. And a couple things became. Comparing from the beginning is like, yes, there was high agreement, but there's also some degree of disagreement. And in those cases where we disagreed, what we basically saw is that often the AI was picking up on some global trend that maybe a moderator was missing. Right. And so it became there was a meeting where I had with a customer where I had to report, hey, the agreement numbers are pretty low. But we checked the individual examples and we start siding with the AI and realized, oh, it caught something. And so there became what started as a report card of this goal. You should mimic moderators. Ended up becoming a learning tool for moderation teams. These are things you might be missing that AI is catching.
- [21:38]
A
Okay, I saw this in your application and this is literally why I invited you on the show. I am working on a product that has an exact analog to this. So I'm building like AI synthesis of customer interviews and the product team can actually edit what the AI produced. And so you would think that's a good feedback loop. But not all product teams know a well defined opportunity is. And sometimes the AI is better. And this is like, I've heard this a few times from different teams on the show. Now it's like we start with this benchmark of we want to get as good as the humans that are doing this. And then we realize like, AI might have the potential to be better than the humans that are doing this. And there's an obvious domain of self driving cars. Like, I think we are probably already past the point where self driving cars are better than human drivers from a like, accident standpoint. But I think this is like an uncomfortable area for like humanity in general. What do you mean? This is going to do my job better than I am. But also like, for us as product teams developing the products, like Dan, you started with. We started with a holdout set. It's like you have your test set where you know the answers, but you don't know the answers because the human response may not be perfect. I'm really curious to hear how you manage this, both from a data science standpoint of how are you even scoring your AI, but also from a customer management standpoint of how are you communicating to the customer that, hey, our service might be doing a little better job than your team?
- [23:19]
B
There's a couple of things to unpack. I think one way that we do it is just exactly like Dan said, we understand what spam looks like just by interacting with the data. So you start to get this mental model and you can just do an error analysis and get a sense of, okay, this is wrong, but that takes time. So a way that we automate that is by using AI as a judge. And what you can do is if you have, if you know what good looks like you can describe what that is. So in our case, we can use a customer's policy. So like we'll say this is a lamina call on the platform and it's not. You pass that to a reasoning model and you include the original content plus the AI decision and the moderator decision and it becomes a tiebreaker. So I think that that could might be work for your product too. If you have a sense of what good looks like, you can create some sort of eval or a skill and then judge against that.
- [24:13]
A
So you're basically using like an element as judge to referee between the human label or the human response and your agent's response.
- [24:23]
B
So yeah, we do that. I will also say the model's pretty robust though. So as long as moderators among themselves agree enough, then even like a certain amount of disagreements between moderators, like the model is pretty robust to that. So we don't actually need to have very high agreement rates to still perform well. So it's almost like Dan was saying, it's like this, it's this fuzzy metric that we use. Like we don't want it to be really low. That would probably be a sign that something's wrong. But we've also just developed a tolerance where we know it's like there will be disagreements, but that things are still healthy in productions.
- [25:05]
C
Nikki, weren't you telling me there have been some clients who you've had to go back to and we've actually had to pause some efforts sometimes and they go back to their trust and safety team internally because there is so much disagreement between human moderators and so they're like, oh, wow. Yeah, I guess our policy was unclear there and we're actually helping them find kind of gaps in their policy.
- [25:24]
B
Yeah, that does happen a lot. So we do have things that are more LLM forward where it's more like moderating content and you describe what your policies are and it does really seem every time you have to sit down and talk about your policies in a way that's concrete enough or an LLM to understand what that boundary is, it opens up all these gaps. Okay, actually what is that boundary? And teams have to go back, they have to look at different examples and it does create this. Yeah, this process where it's like a self audit of your policies or like they sound simple and it sounds straightforward, but when you get into the nitty gritty, then yeah, you realize it's not clear at all.
- [26:07]
A
This is actually one of my favorite parts about building AI products is that so I historically teach Humans. I run a training business and I run I design courses and my courses all have rubrics. And I started to build, like, AI coaches. And what's great about having rubrics is that's exactly what you need to give an AI to learn a thing. And then you, like, see what the AI does. And you're like, oh, my rubric isn't as clear as I thought it was. And then you're like, oh, that's why humans don't quite get it either. Like, maybe I need to fix the rubric. So I have taken to like, the data science part of AI building like a fish to water. Like, it's just so I didn't know all this existed. Like, I didn't know this is what ML engineers did. But I like, literally. And there's so many parallels with dealing with qualitative customer data. There's so many parallels with teaching and like, clarity around rubrics and models. And I just am like, I love every bit of it. But not everybody thinks that way. Not everybody thinks that structurally. And I think it's really hard for people to be like, no, it makes sense to me. What do you mean the model thinks about it differently? Or even somebody else on my moderation team thinks about it differently. Are your, Is this mostly a net positive for your customers? Are they looking at this and going, oh, wow, we really can get better, or do you have to do a lot to handhold them through that and be like, it's okay that you disagree. This is normal. Like, it'll get better. As we refine the rubric, I'm just curious about the, like, change management part you might have to go through with customers.
- [27:41]
B
Yeah. I don't know about you, Brian and Dan, but I found it to be the opposite where customers, we see their disagreements and they're like, thank you, thank you for showing these because I think a lot. So I think we've talked about a lot, like, the moderation. It takes a lot of effort and sometimes these are. They'll hire contractors to do this and they have QA processes, but if they're pretty manual, what will happen is they'll take 10% of people's decisions and then have a, like a QA manager also give the same decision. And then they'll, like, they'll look at accuracy, but that's a pretty intense process. And there are a lot of inconsistencies. There are inconsistencies between, like, we're making human judgments. There's always going to be inconsistencies. So this is a way where we now have this objective, neutral measure of what the decision could be. Customers know that they have inconsistency, inconsistencies in moderation and when we identify what those are, they're very thankful. Like I'll give you an example. I was working on a project for a dating app and the machine learning results like just weren't doing great and in fact they were really nice. And then we started to regress a little bit so I had to dig into the labels and it turned out this one moderator was just going off on their own and like the, like making different decisions than the rest of the moderator group. But some of the same spammy looking accounts were banned by most moderators but approved by this one moderator. So that was something like we could just find like this moderator disagrees with our model more often. So I let the customer know and they're like, okay, great. And then they followed up and like they could give that person training to make sure that they were more consistent with policies. So yeah, I've noticed customers are always like very welcome to those things because they know it's a, it's an issue that they don't have a great handle on or it takes a lot of effort to have a great handle on.
- [29:47]
A
Yeah, in a lot of ways it's, you're creating like a self improvement loop for their moderation teams which they probably have been really hungry for. They just. How do you get there without something that can scale? Like AI can.
- [30:00]
B
Yeah, exactly. Right.
- [30:02]
A
Okay, let's talk a little bit about current day. So it sounds like early on you were able to combine more traditional ML methods with this new. Let's get even more granular, let's get a little more qualitative by adding some intelligence models. What does today look like? Okay.
- [30:23]
B
Okay. Yeah. So really we've just been building on that pattern and the way that I think of it is we just have more models that meet different use cases. So what Brian was saying, some cases we want really low cost because we want to deploy something across the whole platform and we want to cast a very wide net. So we need really low cost and latencies. So for those we might use like LLMs. In other cases we want something that's adaptive, looking at the whole profile. And we basically have this menu now of different models fit for different purposes with a layer of tools stacked on top of that to help with the end to end process of going from the definition of what you want and don't want in your policy to launching something in production. And then after you do that, monitoring and then making sure that things are still on the rails, and then that takes a lot of different forms of what those tools are.
- [31:25]
A
Okay. I realize, as you were describing this, I'm not even sure I want to get a little bit into, like, how you even onboard a new customer. Because I realized, as you were describing that, Nikki, every company has their own policies and they have their own content types. Right. So, like you mentioned dating apps. I imagine you're doing content moderation on profile content, but also maybe even on messages and maybe even on. I don't know if there's any public content on those sites or if it's all private. Whereas that looks very different from, like, Twitter, where almost everything's public. So tell me a little bit about, like, when you're starting to work with a new customer. What does that onboarding process look like?
- [32:07]
B
Yeah, I can give, like, a brief overview, and then Dan and Brian, feel free to jump too. We start by really understanding the problems that customer is facing and, like, what's failing right now? Like, why are they coming to us? Like, what? And sometimes that will look like they have other vendors that have these, like, more like black box models where you can get a score. Like, this content is 0.7 on sexual and.3 on violence, and then you can set thresholds on that. But they're finding that they have so many false positives, so it's like flooding their moderation system. And it's like, a lot of the content's fine. In other cases, it's the adaptability. So, like, they might have some system, but then it quickly goes out of date as spammers trying to get around it. So their main core problem is that they need something that adapts to changes in attack patterns. And we do something we call reverse demos, which is we have the customer show us their moderation system and show us, like, what's hard about your problem. And then we do some solutioning where we have an understanding of what models work well for different contexts, and we propose what solution would look best for their specific content, the specific problems on their platform. We're working now to make that a flexible system where you basically just send us data, and then we can config what that looks like, what models we use, how they stack on top of each other, how we transform labels to give you just exactly what you need back so that you can feed it back into your moderation system. And then that, I think, will take out some of the work that we need to do on the front end of really figuring out exactly what solution they need and then building it for them where it can be a little bit more. You give us data, we'll set you up with something. But if it needs to evolve and you want to add in additional models or different layers, then you can do that easier.
- [34:06]
A
You're triggering a few different threads I want to pull on here. The first is your comment about most teams start with these generic scores made me immediately think of these off the shelf evals which maybe you can start with that, but you're not going to get very far until you do your own error analysis and figure out what's unique to your content. The second thing that was in the back of my head was this trend of forward deployed engineers. It's almost like you're describing, like we're almost consulting by looking at their data and then choosing the right solutions. I'm guessing two to three year old company, you're not developing bespoke solutions for each customer. Is this where your like toolkit idea is coming from? Like you have a wide variety of products you could deploy and you're using this reverse demo to figure out like what's the right mix for this customer?
- [34:59]
B
Yeah, that's right. It's part that the toolkit approach, it's also part of it is when we're talking about those models that we're training, those are custom trained based on customer data and that gives us a big bang for the buck like performance wise so that we can, and Dan's done so much work on this, we can have a general training pipeline and a generalized interface of what it looks like where we get data in, we give back like a risk score back. But we can adapt to different customers data by training specifically on their data. And that helps more than some of these black box models. Where you get one, there's one model that's going to tell you the violence score across content for many different customers. And as you exactly say. And I'm just, yeah, like even within like our dating customers, every platform has such different rules of what's allowed and it's not the same way. It's not in a way where it's like this platform is more lenient and this one's more strict so you can have a threshold slider. It's more like there are all of these exceptions that those black box models just can't. They're just not flexible enough to address.
- [36:18]
A
So you've mentioned a few times you're training models and you're training A model for each customer, it sounds like. Let's get a little bit into what do we mean? Like people throw around, they train a model and it means 100 different things. So let's talk a little bit about a new customer signing on. What does that look like? What do you mean by training a model?
- [36:39]
B
Dan, do you want to.
- [36:40]
D
Yeah, I think so. Maybe taking a step back, I can talk about a couple things here. I think for us, training a model kind of basically looks like first you send us your data, we need a snapshot of activity that's happening on your platform. And basically you give us examples of this is what we would have done or this is what we have done in the past to moderate these examples or take down fraudulent examples. So it really just starts with that. And then when we say model training, we literally mean doing the full end to end from that data training, machine learning model or training and fine tuning a large language model that essentially is able to make those decisions of like an accuracy and performance degree that really meets that customer's part. And that really changes depending on the customer and like what the costs are of a mistake. And it can obviously depend on how critical the scenario is, but that's actually literally what we do. And you can imagine like when you hear that sound like, oh my gosh, you guys are just like a bunch of data scientists going like crazy on people's data and that's. It sounds like that. But I think if you imagine model training, a lot of people think maybe you'd use Jupyter notebook, for example. That's like the first thing that people would go for in terms of training the model. And we could do that. We do a lot of work in Jupyter notebooks in terms of investigations. But really what we've strived to do is create basically like a model trading pipeline and platform where essentially we can use reusable blocks that we know ahead of time. Like we're going to have to pull data and it has to be fresh at a regular cadence. We're going to do some sampling, we're going to have to do some data cleaning, we have to do certain training parameters, we have to stack certain models, we'll have to do evaluation, we'll have to do model deployment, et cetera. And so we've just essentially built like a bunch of building blocks that enables us to actually do that in a repeatable way. Yeah, and I think that's where we focus some of our efforts so that we're capable of getting that custom performance while still Being a small team and actually having like impact and being able to do this and manage this for a group of clients.
- [38:35]
A
Okay. So I can wrap my head around this from a machine learning standpoint. I can very clearly visualize what a training model pipeline would look like for a classic machine learning problem. I think when we're talking about LLMs, it feels a little bit different. Unless we're talking about fine tuning. Are we fine tuning the model? This is where how we talk about training a model is sloppy. I'm just curious about, are you starting with an open source model? Are you literally starting from scratch and you're just creating small models for each customer? What does that look like?
- [39:08]
D
Yes. First off, to be clear, for most circumstances with LLMs, we're fine tuning. And obviously fine tuning is so much more analogous to traditional machine learning. Right, because we're actually just fine tuning some of the last layers of that LLM as opposed to going throughout the entire thing. But we've actually found some cases where it helps us to do an end to end train on a much smaller model, obviously because of GPU constraints and things like that. We found scenarios, for example, where let's say it helps in some intermediate representation to create an embedding of a piece of content. An embedding is just a numerical representation of content of an input text, for example. We found it actually helpful to maybe fine tune a model that's capable, for example, of taking instruction. So for example, as you embed this, focus on whether or not it's classified into these categories. And so like with that fine tuning or sorry, training, actually we're able to train a model that's able to take that instruction and we found that to improve performance upon classification later on. And so yeah, those are the couple ways. But yeah, I do think the full end to end training thing is not what we focus on for most large ML.
- [40:13]
A
Okay. And then I'm curious about just cost. I can see how what you're describing could definitely lead to higher quality. But I also could imagine your service might have to be way more expensive than this off the shelf scoring, which may not be a problem. Let me see that. With evals, to do your own custom evals costs a lot more than just hiring Braintree to use their off the shelf evals. But I'm curious, one, how are you managing costs? Is my assumption right there. And then two, is it, is that okay, are there customers where this is just such a huge pain point that they are willing to pay more for that?
- [40:52]
C
Yeah, I think I can Jump in here a little bit. As Nikki mentioned before, we do have customers who are far more price sensitive. Every customer is price sensitive, right. If you tell someone you can deliver the same results for a dollar as opposed to $10, they're going to take the dollar solution every time. But the way we've handled it, at least that I've seen and Nikki, feel free to jump in. Is okay. We just have a realistic approach with customers and say, this is what we're estimating for your volume. If you go with this model and this approach, we can custom train a model or as Dan was saying, fine tune these models. Some of the generation of models we're working on right now probably are only available to an enterprise client just because of the cost to, to train and implement it, but then also just the inference cost itself. And that's a lot of what I've been focusing on now is just benchmarking for these latency sensitive clients. We can technically make like any open source LLM work, but you're getting maybe 10 requests a second. And so if you're thinking about a business that it could already afford, that they're probably at a scale of we have clients in the 10 to 100 million a month range or more. That's not really going to work. You're quickly going into GPU budgets in like several hundred thousand dollars a year no matter what platform you're on. So it's really just okay. We've spent a lot of time experimenting with, okay, here are these model architectures. Here's some tricks and like short circuiting we can do to help. We'll f, you know, we'll lose a little accuracy here on, on the evals, but we'll gain a ton of performance and throughput. And it's really about just talking with the customer and figuring out like, hey, if you want to, you can't have everything. But you tell us what you prioritize and it can be context that you're aware as well. Oh yeah. On this part of our platform, like we really care about accuracy and it's worth the cost, but on this part we're okay. And like it's a little bit of an edgier zone maybe in our domain, but so it's really just like flexibility and benchmarking these numbers and presenting it to the client and tweaking the various knobs. We have to figure out what works for their budget and their business, I would say.
- [42:54]
B
Another thing I'd add to that is a lot of times, if things will eventually fall to a human moderation team. And that's more expensive, too. So for a lot of our customers, they might be spending, depending on what content's being moderated, it's only like $0.15 per message of, I don't know, like $0.50 per account profile or something like that. So even though it's like, it depends on what the domain is, Brian's saying, but if the alternative is to have a human look at it, then, yeah, we usually come up ahead in cost.
- [43:30]
A
Okay, great. Yeah. Okay. So what it sounds like to me is there's this really. We're in this really interesting space where I think AI is helping companies move away from this. Like, in the SaaS world, we build one product that everybody uses the same way. And I think we're starting to move into this realm where software can be a lot more adaptable and more custom for customers. But we're not moving fully into, like, bespoke consulting or bespoke professional development. And it sounds like your company is in that sweet spot and that you're trying to build out these tools that you can mix and match and even train models that might be bespoke to a customer, but in a way that's like, platform driven. Tell me a little bit about this is an area that I think is just so new to product teams and thinking in this way. Tell me a little bit about just how you guys think about that. How do you make sure you're not falling into this, like, bespoke consulting trap? Like, how do you productize this? How do you think about that?
- [44:33]
D
One thing that we're actually focused on is just giving customers the tools that we would use to do that. Right. So just empowering teams in general. Oh, I find it helpful to have these building blocks. I find it helpful to have this eval platform, this observability. Let's just give those tools to customers. And so that's been our philosophy. And I think a great example is in some of our, like, more LLM focused products that are basically about moderating pieces of content, whether it be any type of media you can imagine. I think Nikki was touching upon this. But there's this example of, like, customers having to do a couple things, which is say what they want in terms of a policy of what's allowed and what's not allowed. There's a whole discipline to doing that properly as well as, like, evaluating after they've given this instruction to an LLM, like, how well does it do? And so we're building out entire basically, like, test suites and optimization suites where customers can do this themselves. Right. And that's one of the core things that we provide the folks. And for sure there's some initial onboarding where we're introducing the concept and walking through of what it actually takes to make this performant. But once customers are up and running, we find that they're super successful on iterating on their own and launching these things to have this LM be launched as a moderator or like a fraud detector on their platform. So I think that's one of the philosophies that allows us to do this customization. It's just giving folks the power to do it themselves.
- [45:49]
A
This is also a theme that has been emerging. So I've done a lot of interviews with teams that are building like customer service agents or sales agents. And one of the ways that they do their own evals is instead of trying to eval across all these different customer contexts, they're basically giving their customers access to the traces and they're giving them like an annotation tool that then feeds into what they're using for their failure modes. And it's like they're pushing all of this to the customer, which makes so much sense. I'll share. When I first learned about evals and I learned like we're using code assertions and elements judges and like, my first thought was like, why isn't the customer the judge? And what we're seeing is we are now getting to the point where the customer is becoming the judge, which I absolutely love. Nikki, you wanted to jump in as well?
- [46:41]
B
Yeah, I think damage is a really good one also. I think we've. I don't want to be like two rosy. Oh, we create this flexible system from the beginning because I think the truth of it is we build something that we think will, you know, meet core use cases. And then every customer is different and every customer teaches us, oh, there's actually this that's needed. And it's been like, I think this push to make things more flexible has been a little bit more out of necessity where every customer needs something different. So we need to approach the space where we can swap things in and out and have additional branching or condition logic or something like that. So I think what we normally do in practice is we usually come across some sort of situation for a customer that are product or solution suite can't handle out of the box. And then one of my, like, my favorite meetings is when we have a cross functional technical meeting. Okay, how can we solve this? What are all the Trade offs what makes the most sense, do it specifically for that customer to meet the need. But then we understand, okay, do other customers need this? And then how do we generalize it? And we've done that pattern I think now a couple times across different tools. And I think it's working for us. I think it would have been like maybe we would have designed things differently from the beginning. But it is nice where you get. It's a little bit hard to know what is really specific to this customer versus what's general. But this customer is the first. And then that pattern helps us figure out that it's more general and worth building something.
- [48:22]
A
I think this is the perennial product management challenge, right? Every B2B company has their like one off deals for the large whale customer that just needs. And it's really, you gotta figure out is anybody else ever gonna use this or am I just literally building this for this one customer? And what's funny is like it's easy to say like you should just not do that and you should only build things that generalize. But that's not true either. Like you have a large customer and you can't just walk away from that. And probably something generalizes. And so I think this is like one of the most fun things about one just software development getting cheaper. But also like having intelligence built into our tools allows us to build in more flexibility from the beginning. And I think we're going to see this like platform toolkit, ecosystem mindset, I think becoming more and more common because it allows people to have this nice hybrid between. We're almost selling custom consulting, but we're servicing it with products, which I think is amazing.
- [49:25]
C
Yeah, I'd be super interested to hear we've been figuring this out internally on our own, but because the cost of engineering, as you said, has gone down so dramatically, we've got people, literally every single person at the company has made a vibe coded prototype of something for an idea they've had, which is awesome. It's just so cool to see people embracing this technology and taking their ideas of how to make the product better to life. But I'd love to know like what you've heard from people and your own building, like how do you handle. I feel like a problem we run into sometimes is like we have multiple competing prototypes on not the exact same domain surface, but like a shared domain surface. And then we also just the speed at which changes come out. We, we sometimes like we're as Nikki alluded to, we have these weekly syncs where we're demoing and we're talking about the toolkit solutions that we have, but would love to hear what you. What you've encountered and seen work.
- [50:21]
A
Well, yeah, it used to be we had product managers to manage the bottleneck. That was engineers. Right. It took a long time to build, and so we needed somebody to say, what should we build now? It's easy to have ideas. It's easy to build those ideas. I think the bottleneck has moved, which is what should we put in front of customers? And I actually think that's still a hard. We cannot put. I know a lot of companies think they can do this, but we cannot put everything in front of all customers. Now we have one of my favorite models, which there's not very many positive things I'm going to say about Meta, but I will say they made one thing that I think is we're going to see way more companies do, which is they built a system where literally any engineer can push to production, but it goes to a fraction of a percent of customers. And they measure the impact, and if it looks like it's having a positive impact, they grow the percentage that it goes to. And this is fully automated. And this is a way. This is one way. It's probably not the only way, but it is a way to test should we be rolling this out to our customers? And as long as it's not always going the same tiny fraction of customers, we're not fatiguing our customers. And I think this, whether it's these automated systems like this, whether that's the role product managers start to play, I'm not sure what it's going to look like. It's probably going to. We're probably going to have lots of solutions to this, but I don't think we can just build anything and everything and put it in front of our customers. I think companies are going to try that and it's not going to go over very well. So I think there's still a bottleneck. We're just still learning how to deal with the new point in the system.
- [51:55]
C
Yeah, it's almost like you need this curator of things, which is why people
- [51:59]
A
are talking about taste. But I hate that. I hate that framing of it. I don't care about your taste. I care about my customer's taste. That's really my issue. There's. Okay, so we've touched on evals a little bit we talked about. You're pushing as much of this to the customer. You're giving them the tools that you yourself would use. Let's think about this from a toolkit standpoint. If you look across your toolkit, how do you know what's working, what's not working, what needs improvement, how are you iterating on especially I know evals apply to, I mean really came from the machine learning world, so it applies to your machine learning models. But I'm also really curious from an LLM standpoint, what are you doing to just make sure that as you evolve these products, they continue to work for everybody and not just over indexing on one customer over another?
- [52:50]
B
Yeah, I can go in more of the LLM side. So we do something which I think is very common where we'll have customers create a golden set. And that is just a collection of examples that they tell us, like how it should be moderated. So it might be like kill yourself and that should be violence or maybe like bullying, I don't know. So then we will take their policy, run it against that golden set and then understand where is it missing things, where is it finding things that it shouldn't? That's the core that's built into the product from the very beginning. And then we do this tight iteration loop where you update your policy, you test it in on the golden set and, and we can, we can actually automate that. So we have this policy optimizer that's like this agentic flow where it will do an error analysis. It will suggest changes to correct the labels in the golden set. Because we sometimes find customers. Yeah, labels are wrong. And we're like, is this like even
- [53:49]
A
in the golden data set?
- [53:50]
B
Yeah, it's like conflicting your policy or it will recommend changes to the policy and it's still a human in the loop. You're at the helm. You can decide if you want to accept or reject the recommendation, but it will create this loop and it will try to optimize your policy against the golden set. So that's core, but now we're working on developing additional things above that where you're almost getting close to evaluating the eval. So I'll just give you some examples. And this is very much in this toolkit framework where one is you want to compare policies to a different one or like a previous run to see how things are improving or aggressing. You might want to take your policy, run it against eight different Frontier models, like Frontier LLMs, because they all are going to judge things just a little bit differently. But then that also helps you understand, okay, for a given accuracy level, what's the cheapest and fastest that I can use. And then there are other things, like analyzing bias. So one thing that we're looking at is this way where we want moderation to be fair across people. You don't want it to be like, unfairly judging some groups more harshly. So there's a way, like in that golden set, you can modify things, like you can change the language from English to Spanish, or you can change references to race to one to another. And then you just see, is the model still consistent? Other tools. Looking at the golden set, we talked about golden sets, they're really sometimes silver sets. They're not very good. They need some work, and they're like metrics that we can do to understand. Okay, is this representative of the policy? Is it representative of production data? Because sometimes you might just be missing large swaths of your production, and if it's not in your golden set, then you're not testing it in your policy.
- [55:42]
A
This is going to be my first question for you, which is you don't just create a golden data set once. Right. It's something you should be curating over time. It definitely should, at least on some level, be representative of what you're seeing in production. I love the idea of pushing this to the customer for so many reasons. One, we get to take advantage of them doing the work instead of us doing the work. They are the domain experts. Like, they should be deciding that. But there is a data science skill here that I don't think we can expect our customers to have. It sounds like you're trying to address that by evaluating the data set itself and then giving the customer feedback on that. Did I understand that right?
- [56:26]
B
Yeah, that's right. One idea that we're playing with, and I think we're going to expand into more, is giving tools to the customers. If you want to dive deep, you can. But also giving those same tools to an AI assistant in the app with the. With the description of skills. This is how you should approach these problems. This is how you should do a model comparison. This is how you make sure that the goal inside and your policy look healthy. And then that kind of takes some of that burden off the customer where, like, really they can just describe their goal. I want to get this ready so I'm confident to launch in production. And then the agent can be like, okay, to do that, we need to make sure that the golden set is representative that the policy. I can go through that and offload some of that mental burden.
- [57:11]
A
Yeah, I've talked to a few teams now, too, where this is exactly what they're doing. They're creating an agent coach for their customers to learn how to do the data science part of it. I'm curious. This is such a little minutia thing, but I've had this experience with working with Claude code. I can't tell you how many times Claude code makes a suggestion that basically would lead to data leakage. And I have to just be like, claude, you can't use that example in my prompt. What are you doing? You just pulled that from my test set. What? And just there's all this. And I'm new to this. I learned this stuff, like, all in the last year. And so I feel like there's gotta be so many people out there, and especially when we push this to customers. We can't even rely on models unless they're trained for this, unless they're instructed for this, unless they're given the knowledge for this. Because Claude knows. When I push back on Claude, it'll be like, oh, you're right, that's data leakage. We can't do that. Let's do this instead. But I have to know enough to tell it. And so what you're describing feels like a product in and of itself. We have to build a product that's a good data scientist that not just as a good data scientist, but as a good data scientist that can coach our customers. Is that the path you're exploring?
- [58:29]
B
You're totally right. We're exploring it. And you're exactly right that it is a tricky problem. And even we ran into the exact same issue. So in that policy optimizer that I was telling you about, we'll find that on real customer data sets, it gets over eager and it will change your policy to really overfit onto the golden set so that it gets it right and we, like, instruct it, simplify. So we're trying to find ways now, like, you have to build and guard rails. And one thing that we're experimenting with that one is have two passes. Like one where it makes a recommendation, another one where it takes that recommendation and like, checks itself. Is this something that could overfit to the golden set? Is this so. Yeah, we're experimenting on that of how. Yeah, because we don't want to lead our customers down the wrong path. That's the, like, last thing we want to do. So, yeah, it's. That's becoming like a whole another thing that we're trying to work on. How to do it the right way.
- [59:27]
A
Yeah, yeah.
- [59:29]
C
I think I'll just jump in and yeah, just say I think I also heard the data leakage, right, because the. Our customers are a business, but their end users oftentimes are real humans with real lives. And as some of our customers are like social networks, like dating apps and whatnot, and like super sensitive information. So we're super cautious about introducing, like, slapping it in there in the product, as you said. And so I think we want to move even faster and we want to get. I think what I was hearing from you earlier, Teresa, is there's this complex product. I think I'm going to pick on Dan and Nikki a little bit here. When I got here, like one of the first days I was here, there was a confusion matrix that Nikki was showing and I was just like, what the is a confusion matrix? And like, I think about people who are already starved for time in their workday and like, coming into our product and it's real. It's like there's cool stuff going on there, but I don't even have the energy to finish my next moderation task, let alone learn data science. And, like, I think what we're really excited about is this potential for an AI assistant. That's, as Nikki was saying, I know what I want to do. I don't know what knobs, I don't know what things I need to do in your product. And so just chat with me. But we need to do that in a safe way. We need to do that in a way where, like, you can roll back changes, and we need to do it in a way where we're not going to leak data. And so that's. I think that friction, a little bit of what you were talking about earlier too, is just what do you actually show to customers versus, like, what do we play with external, like, internally is maybe what people call taste.
- [60:55]
A
You know what I love about what you're saying is that the reason why pushing evals to customers really resonates with me, if I put like a content moderation response in front of a customer, almost every human on a content moderation team can say, yes, this was good, or no, this was not. And if they say, no, this was not, they probably can verbalize why. And at its core, that's all we need, right? Just yes or no and why, and all the rest I feel like we can do for them. And this is the part, like, when I first learned about evals, I was like, this is where evals are like the new discovery habit. This is the intersection. We need to learn how to build interfaces and experiences where we can just get those Two pieces of data, yes or no and if the answer is no, why? And that's the engine. Right. That's our feedback loop. And so I love that teams are pushing for this. And yes, to Nikki's point, it comes with all these super hard challenges and representative sets and high quality sets. And are you going to do this one time? Can we get you to do it over time? Because we probably need to make sure your set stays representative. But I think we're going to see a lot more of this because it turns out like all of our support tickets are our customers saying this is wrong and here's why. So it's like we just have a new mechanism that hopefully feeds into our models and helps to train them.
- [62:15]
C
Yeah, I love what you're saying about like in the background. One of the ideas, I think Nikki, when we were experimenting with kind of latency models earlier this year was instead of just exposing a huge long list and table to an end user of hey, is this correct? Like almost trying to gamify it and like using signals on the back end. Dan was mentioning earlier we've got a tool that helps visualize embedding spaces. And so if we just look at the dist between something we've had a human moderate versus only have had a LLM as a judge moderate. And like that distance is really large. Like maybe that's like the top of the queue and just a product that can really break down like small chunks of hey, here's five things you can do to make your policy judgment better today. Like that's. I think that's something anyone can manage. Right? Like you come in, you click boom. It's like doing a wordle or something.
- [63:03]
A
It's just adding to your moderation queue. Right?
- [63:06]
D
Yeah.
- [63:06]
A
Here's a few things you can do. And by the way, if you do these, your queue will get smaller and smaller over time.
- [63:12]
C
Yeah, yeah. And I love, there's products I think that do a really good job of rewarding user. It sounds so dumb, but like Google Maps. I was in San Francisco and I was looking for something and there was like, I don't know why it was an H Vac company, but it was an H Vac company and for some reason it was on like Catalina island and whatnot. I was like, there's no way this place is on Catalina Island. There's no way an H Vac company is on that. That's just not realistic. It's like I made a small like correct correction, the statistic correction. And every year I get Some like, little piece of feedback that's, hey, you helped a hundred people do a better job here.
- [63:44]
D
Yeah.
- [63:45]
C
And it's just like those little things, they're so stupid. It's like just this little email, like, it doesn't really matter, but anytime I think a product can do that and just reward a user for being a good. Being a good patron of the community is just. It just makes you see the value of what you did.
- [63:59]
A
I also think in your context, like, I'm going to go back to this very beginning of, like, my question about rubrics and what if the AI disagrees with the human? We all want this feedback because we have a shared goal of can we make our site, our service, our product better? Everybody on that content moderation team already cares. That's why they're in their job. I think this eval piece is. This is a way for you to increase the quality of what your end users experience. Experience. And I don't think that's a hard sell. I think it's. We have to present it so it doesn't feel like we're doing error analysis.
- [64:36]
D
I think that's entirely right, Teresa. And it's. What's amazing to see is, like, when customers actually catch on to the idea of the impact of their evals and the feedback that they're giving up actually changes the model. We've seen customers, like, really prioritize this. So we gave the example of our models holding out certain decisions and just waiting to see what trust and safety teams do and then doing the comparison. We've actually had scenarios where teams will come back to us and say, wait a second, you're holding out certain decisions. Which decisions are you holding out? Send that to a separate queue. We're going to put that as a priority because we want to see that comparison and we want to specifically train your model. And so there are cases where basically we get that feedback on a daily basis. People are working through this and what that does for them. If they decide on maybe 10, 100 different decisions on a daily basis, our model can then expand and become more confident and make thousands of decisions on their behalf. And they're seeing that feedback loop happen in production. And so it's really cool to see folks changing their moderation flow to integrate this AI service in it.
- [65:35]
A
That's amazing. Oh, man. This had to have been like 2010, 2011. There's this article that came out, it was about cognitive surplus. It was this idea that humans want to contribute to their community. It was about Wikipedia, I think and how does Wikipedia get all these people to write articles and for free? And it's just this idea that I think, like inherently we want to improve things, we want to contribute, we just don't always know how. And I think this is such a silly little. Like, it's not silly for these teams at all, but like, it's such a little. It's just such a. This is what got me excited about evals and about data science in general, is that to me it's the ultimate feedback loop. We're just. If we're going to put all this non deterministic stuff in front of our users, we better have a way to just have a feedback loop and know, are we going in the right direction? Okay, we have three minutes left. Let me ask you this. Is there anything you wish I had asked you that we haven't covered?
- [66:39]
C
What's next?
- [66:41]
A
Yeah, what's next for your toolkit?
- [66:47]
D
Wow.
- [66:47]
C
I should have a better answer prepared. I'm kidding. Yeah. I think I'll just harken back to the agentic workflow. I was a bit of a skeptic, I would say, about agentic workflows, but even from a surface level of kind of hiding the complexity of the product from an end user and just letting them use natural language to, to really interact with the system like that to me is like a clear win. But just this idea that like I can go and say, hey, AI, you're really good at monotonous tasks. You may not have the taste that we talk about, but like you can go on a loop and tell me the numbers and run some like, primitive analysis and like I can go do things that I need to do for my job, like outside of that, and then I can come back and I'll review the results with you. So, like, I think we're really excited about this world where we have this toolkit where it's like, you come in, you spend 15, 20 minutes with us and like you help us do your job, but not your job, but like you help us do your moderate your content and enforce your community guidelines better and then go do the things that are important in your life and your business and then come back and we'll have done a bunch of work for you. But as you said, human in the loop, like, we're not going to push those changes live. We're just going to make suggestions and help review those suggestions with you. So that's what I'm super excited about, is like doing that in a safe and empowering way.
- [68:06]
D
Yeah, I Totally agree with that, Brian. And I think some of it is actually striking the balance of basically knowing what to make agent tech and what to actually keep people in the weeds on. Right. So a lot of our tools are actually rather innovated that there are some places where we want people to be in the weeds. For example with data. You mentioned the example of us visualizing a platform's network activity and their moderation decisions across a visual map. And that's one where we've been opinionated on like you should see what's happening on the platform. Right. But we've also found that as you're visualizing the entire map, it can be difficult to sift through and understand what are actually the emerging trends. So I think to your point, exactly, we're trying to integrate like more agentic flows in some of those experiences. Or maybe they'd also get maybe an alert saying hey, there's a new emerging trend happening, there's a new scam attack, et cetera. You should be aware of this trend and here's how it looks. And so it's pairing these in the weeds investigation tools with these outcomes based results that are more of that agentic experience.
- [69:04]
A
I feel like that's such a data science mindset.
- [69:08]
B
Right.
- [69:08]
A
Of give me the big picture, let me zoom in, let me go back and forth.
- [69:12]
D
Yeah, you got us.
- [69:15]
B
I think the two things that I'm most excited about. One is on the machine learning side. We're great on observability and we were talking about those holdout sets. We're looking at agreement. We have a really good sense on how the model's doing in production and we are trying to now set more of those things up for these models based on LLMs that are deployed in production. Getting that feedback loop of are they working well? Understanding like when data is shifting and just getting like a more like robust and I guess like right and down. What you're saying something more where you don't have to just go in and check the system all the time, but there's something a little bit more proactive or like you're alerted if something looks off or if like the distributions or like your clusters are changing. The second one is around. We're talking about like the orchestrator and like the workflows. I just think that's going to be so powerful because there are so many different customers who have like foundation. One is to send content to some model and like maybe it's traditional ML, maybe it's an LLM and you get a response back but that doesn't work in most cases. Usually there's something where maybe you went to analyze the content first and then escalate that for a holistic profile review. If it's like bad content. Or maybe you have some sort of company where you have different moderation rules for different contexts. So like maybe you have like different forums that all have different rules or different like different experiences that have different levels of roles so you want to route your content to the right policy. And there's all these. We come across like dozens of these use cases. So I'm really excited about adding in that flexibility where it doesn't have to be an engineering team on our customers side who has to create this like complex integration. They just send us the data and then the trust and safety team can configure it however they want and build up this very flexible, powerful system all without relying on, on engineering. Like no, no shade to engineering. But like we find that's always like a key bottleneck for our customers.
- [71:30]
A
Yeah. And I that this is the other piece that I feel like these tools are starting to unlock is that we're seeing non technical people design workflows. Yeah, that's pretty cool. And like you talked about this like pipeline for your training models like we're more humans are starting to learn these concepts of if this, then that and this happens and then that happens. And like I think we intuitively, like most humans intuitively get it. They just can't always translate it in those words. And I think we're seeing more and more interfaces that are helping them think that way and helping them. I really think about it as like we're all learning how to collaborate with models. That's exciting. I'm excited for what that unlocks. Okay. We are overtime and I definitely want to respect your calendars. Thank you so much for sharing your story. This has been really delightful.
- [72:23]
B
Yeah, it's been great. Thanks for having us.
- [72:26]
D
Thank you.
- [72:27]
A
If you enjoyed this conversation, please subscribe in your favorite podcast app and give us a rating as it helps others find the show. Thanks, I appreciate it.