B (3:43)

Well, it's now recorded for posterity and it's a pretty benign black hat introduction to technology really.

A (3:49)

Yeah, I mean, I mean, you know, it was, it was quite a fun challenge because the hardest thing was trying to make the bots not look like bots. So introducing a degree of randomness into the pathing was quite important because otherwise you would have an auto detection by, you know, whatever systems they had on the servers at the time. So it was, you know, it was intellectually interesting to see that there was a lot of benefit to injecting random noise into kind of, you know how I thought. Anyway, it's a very random story but.

B (4:30)

You know, well, when, when you weren't involved in that intellectual pursuit and, and of course this is years later because you were 14 when you were doing that. But how did you get so obsessed with marketing measurement? Because it sort of feels like you're a Roman Empire in a way.

A (4:47)

What do you mean by the Roman Empire?

B (4:49)

Oh, do you remember that whole meme? It was like, I don't know, a couple of years ago where people were referring to things as their Roman Empire, like their obsession.

A (5:02)

Yeah, it's. I think marketing measurement for me is a really interesting problem because it's a very complex and hard to solve problem with not much ground truth that exists. I think the majority of the industry would like to talk about experimentation, for example, today as ground's truth in marketing measurement. Conveniently ignoring the fact that most experiments have statistically very wide kind of bands of certainty in them. So an experiment does not give you a point estimate. And so I think for that reason it's a very, very complex area and hard to understand area, but I think it's an incredibly valuable area. Why do I think it's an incredibly valuable area? Well, so much human capital and investment, whether it be around media, advertising, entertainment is mostly funded by advertising, the distribution of products, the growth of products. Huge amounts of the economy is invested to some degree or another in growth, marketing, pricing, advertising, all those various disciplines. It's also probably because of that lack of Clarity on measurement, in my eyes, one of the least efficient parts of the economy. So, you know, I wonder how much wastage exists in our economy purely because we're bad at understanding what makes companies grow and what doesn't make companies grow. Conversely, I think combine that with a myriad of very poorly aligned incentives. So, you know, I would say that it feels quite patently ridiculous to me that the business supplying the most amount of probably measurement data out to the market also is the largest seller of ad inventory to the market. Only in advertising would we have that ridiculous conflict exist. And so for those reasons, I think it's very, very important both from an economical level to add clarity to that market, to reduce probably the most inefficient part of the global economy is very, very important problem to solve. And on the flip side, I think that there are very, very powerful forces deliberately making that economy less and less efficient by design. So, so I think those are, those two forces make it a very interesting place to play and a very interesting problem to solve. I think the, the kind of most marketing measurement as well hasn't really had much, you know, algorithmic evolution and things like that. Like no one's really considered how do we evolve the algorithms considerably and the sampling algorithms considerably. How do we change the underlying modeling types quite, quite dramatically structurally? And so I think those are all interesting areas to, to play and they're fun. You know, I think as serious as it sounds, it's also kind of interesting when you can start to build models that are able to predict with high degrees of certainty very, very complex problems and start to allocate capital very, very efficiently and start to take the cost of an insight and the cost of answering a question for a business down dramatically. I think that's, it's very fund space to buy.

B (8:33)

I want to talk about how you guys do what you do at mutinext in a sec, but I mean first, can you trust a walled garden? Can you trust their measurement?

A (8:44)

Well, I think you can only really take. I think it's very, very important in anything to follow the financial incentives. And the financial incentives eventually dictate your behavior, whether or not, whether or not that's the intent at the start. Eventually, the nature of the system that we live in in capitalism means that those incentives will eventually dictate your behavior event some way or another. And so I think we have to accept that reality. And so I think you have to accept the reality that, you know, if walled gardens are provisioning information, that that information is probably there primarily to sell advertising so, you know, I think there is some information that, that you can trust, but let's just kind of go through few of the patterns. When we look at market mix modeling, for example, I would avoid using straight impressions alone within a market mix model. Why would I do that? Because the definition of an impression changes platform to platform and also changes over time. So it's not really an apples to apples comparison metric if you're looking at it that way. I think that's like just one example of, of how it can be quite problematic to operate within the parameters purely of walled gardens. I think they're getting better because I think that they understand the requirement to provision data to grow budgets as a whole. Like, I think what we've really, really seen is a past kind of 10 to 15 years was a dramatic increase in the share of advertising budgets. But because advertising as a whole got less effective, the kind of rate of growth of advertising started to plateau against inflation. Like advertising used to grow pretty consistently, like 6.5% many, many years ago. And it started to really taper off in recent times, probably because you can't reliably pour money into advertising anymore and see bottom line growth. What happens, companies then don't invest in it as much. And so I think the wall gardens are kind of understanding that category problem a lot more. And that's why they're doing things like providing MMM APIs for cleaner data sets. I think that's why they're doing things like trying to talk about incrementality testing a lot more, because they understand that there's an imperative to grow the advertising pie or they'll be fighting over a shrinking pie. And I think they all know that's very, very bad for business. That being said, you know, again, I think it's, you know, the walled garden should stick to provisioning data rather than trying to control measurement pipes. And I think it's incredibly dangerous when walled gardens are playing in, you know, partnership programs where they're trying to incentivize certain behaviors in the MMM market or the modeling market, where they're trying to incentivize and, you know, play kingmaker on vendors for certain behaviors. I think those are all really, really bad outcomes. And they're bad outcomes not just for the technology. They're bad outcomes for trust. They're also bad outcomes for marketing's presence in the boardroom. Like so, for example, by the market mix model that's being executed by my vendor and it's paid for by, by my, by my agency, and it's paid for by walled garden, there's a pretty strong argument to say that's a breach of SOX compliance. And so that would mean that you couldn't use that market mix model to represent and advocate for more budget in the boardroom. Like because you don't have the appropriate internal controls to say that that is a robust financial model, that you could reproduce and reproduce its governance and understand, you know, how and why that model generated those results and that those results had appropriate internal controls to govern that that model wasn't fiddled with for any execute, any person executing within the kind of supply chain. And so I think all of those issues are just, you know, underdone. Like when we talk about trusting walled gardens, like I think we've got to go a step further and go what are the implications of the systems that are being set up around measurement by very, very large players and how do they actually start to influence our ability to prosecute the long term agenda of measurement and particularly measurement in the boardroom?

B (13:10)

And that long term agenda is at the very like least just facilitating a conversation that uses the same language between the CFO and the CMO because they bizarrely are at cross purposes sometimes and they really should be rowing in the same direction 100%.

A (13:30)

And I think what's the central challenge for most marketers today? I'd say it's two things. One, we don't have a lot of clarity on what drives growth and what doesn't drive growth. And to get that clarity costs us a lot of money either on consultants, agencies, whatever it is. Yeah. The second thing is that we don't have, we don't have a perception that marketing is a gross driver in the boardroom. And I think that's the tough truth for most marketers. We are seen as a fungible cost. A really good example of this is I was in with a, with a US CMO not so long ago and we had a pre, and we went in with, to their finance director was asking about a budget cut. And you know, and my response was well, how much revenue are you cutting out if you're willing to cut out if you forecast? Because I said here's what's going to happen. So just very happy to do the budget cut but I think you need to make sure that you're reporting the revenue cut. And of course that's a very different conversation. Right. Like, like you know, if you're, if you're very confident saying those things. But in order to get to that point, you don't just need a pretty dashboard With a bunch of ROI charts. You need well governed, reproducible models that can stand up to financial scrutiny and audit. And that is the only way that you put yourself in the position to have that sort of conversation. And so that to me is the central, you know, the central two problems. If you look at the cost to get an answer and the perception of, you know, us being a fungible cost rather than a revenue driver referring contributor. I think those are major, major issues that we need to solve as an industry. And I don't think we help ourselves in. We set up our structures in the wrong way. That in that sense, I think it.

B (15:26)

Is so interesting that of course this is a technical challenge, but it is also a framing problem. It's how we talk about something and the words that we use. And if you frame something as a cost center or if you frame it as a growth opportunity, then the mentality from the start is very different.

A (15:45)

Yeah, totally. And I think it's, you know, for the most part, we know that advertising drives not just incremental sales, but also baseline and long term sales growth. You know, there's plenty of research, both at an individual brand level and at a kind of a more macro level with books like, you know, research I've done by people like Byron Trump and all of his work. There's enough research that has been done where you can pretty materially point to the fact that advertising is a material force in the short term for incremental sales a lot of the time, but for long term sales growth, it's probably one of the larger forces in terms of building mental availability and things like that. It's pretty. Other than price and distribution, you've got really those three forces. Price, distribution, advertising to drive whether or not a product is able to get widely into the hands of consumers. And so on that basis, I just think that we have to get much better at framing the argument that way because, you know, and you know, I love the advertising industry. I think that the advertising industry does great work and I think that we do important work in terms of routing products to market. I think we do a really, really bad job of telling that story into chief financial officers. There's your dog walker.

B (17:16)

My. My dog agrees. A terrible job. You did an absolutely terrible job. That was Oliver. He's made a few appearances on the podcast recently and the dog walker has just arrived. I'm going to use that though, as a segue to start talking about Mutinex because I want to make sure that our listeners know what it is. I mean, I'm going to just read from your website and just on the tin, you guys call yourself an end to end marketing mix modeling platform. So just unpack that for me a little bit. It's a jargon free zone, just FYI. So yeah, what does that actually mean?

A (17:51)

So let's scrap the term market mix modeling because I don't think anyone wakes up in the morning and goes, I want to buy a market mix model like it's not a real product. So I think that what's the core currency thing that we all need when we're kind of faced with a question? You need an answer. The way that you get an answer typically for anybody is I would kind of go look at all of the data information that I can gather. I might talk to a few people and things like that. I would then build my kind of mental model around that particular, that particular problem or answer and then I would, you know, create an argument around that that I could then communicate to others. Right. So that's generally the process. If I'm asked a question and I need to give a response to someone, that's generally the kind of process you go through. In a pretty funny way, that's kind of what we're doing, right? What we are doing is collecting all of the data around a business. The way that we do that is, is pretty unique. I mean, market mix modeling has a notoriously awful experience when it comes to data collection, onboarding, collation. I've heard some absolutely horrific horror stories from customers coming across to us, but we've managed to basically make that process pretty pain free with data os that only allows us to collect the data, connect the data, then we process it through a market mix model. Our central hypothesis on market mix modeling is quite simple. I don't think that it is sensible to build priors, priors being a thing that we inject into a model off, you know, quote unquote, business context. Because what I think business context is probably code for is, is the result that we want to see confirmation bias. So we are testing prize instead for how they generalize across our, our kind of business to understand, okay, oh, do we see this as a general dynamic that can show up across the market and that's really the secret source of that, as it were. From there we, we kind of want to figure out whether or not we can trust the answer. So we, you know, we use our kind of open. We've got a model validation kit that's pretty open source. What is open source that we're using to kind of govern the models and basically. And that tells us can we trust the answer basically that answers coming out of this particular model and if we can't, why can't we? And then from there we then present that in either dashboards that people can really easily interface with, I mean now pretty simple to use, or people can just interface and query the whole thing and it will be able to kind of formulate any kind of, you know, SQL queries and stuff like that that it would need to fire into the database and create, create the relevant answer. I think for us. Like my thesis on this whole space is that it's the cost to get an answer is the fundamental problem to solve. So I don't really think and put much stock in the triangulation theory that a lot of people are pushing around at the moment. I think triangulation is a really, really good way to sell more experimentation budgets for major walled gardens, which is what's happening at the moment. But what I really think the sensible framing is is rather than have three methodologies and try to explain why I've got three different numbers to a cfo, which, let's be real, I know we all want to say in measurement that's the right way to do it. It's a complete thera fee to think that's actually going to work at scale. Like it's not going to work at scale. I think the real challenge is creating MMM as a brain, using kind of experiments to validate, then calibrate where necessary. But I think, you know, calibration should be used very sparingly. You don't want to calibrate. If I've got a junky brain and, you know, so, you know, my, my neurons are misfiring for some reason. You don't just want to point me on the right path and trust that I'll stay there. You want to kind of fix, fix the core problem in the brain, do you know what I mean? And then, you know, long term, I think that then the infrastructure and data collection has to get much easier and the infrastructure to get data out has to get much easier and eventually then the brain, as it were, probably needs to get hands and start interfacing with the, with the, and sending instructions to, you know, the conversion APIs and all that sort of stuff. So I think that's where we want to go. Right now where our business is, is we think we've got the cost of our costs. We think we've nailed getting the cost to get an answer out of these MMM systems and the Cost to operate these MMM systems down to a fine art. We've done that really, really well. I think where we want to get to in the future is an end to end agent that uses MMM as its brain, but can orchestrate a whole range of systems and understand a whole range of things and bring, bring information back to that brain as and when it needs to. And that's, that's where we need to go next.

B (23:25)

Well, before we take a break, I'd love to hear a horror story or something. Two, that you were told by your clients about their MMM experiences. Do a little story time.

A (23:36)

Oh.

B (23:41)

We don't have to use names to protect the innocent and the wicked.

A (23:44)

No, there's, there's three. I will say, you know, I don't want to just sound that we're completely innocent either, you know, because we have created some horror stories ourselves, I must say. So. So, you know, I, I, you know, and I think the only way that you learn to solve some of these problems is by creating your own horror stories to some degree. So I, I don't necessarily want to present Bit Nex as innocent in that regard because we're not. No one's perfect and I wouldn't pretend that we are either. I think two come to mind. One is that we had a customer who had spent a year and a half trying to wrangle data with kind of a pretty major conglomerate. And they spent a year and a half trying to wrangle data back and forth and on a single file. They had over 250 emails back and forth trying to clarify various elements of the file with an analyst. It was a big file, to be fair. But that's just a bit of a nightmare, isn't it really? Like, it's almost just a bit shocking. I think another one that I've seen is I saw a model that had something like 80% of sales of that entire business. It was a fairly significant financial services institution attributed to mediocre, which, you know, I don't know as many. Yeah. The implication there is that if you went dark on your campaigns, 80% of your business would disappear overnight. And I'm not sure that's true. It's a pretty well known, reputable brand as well. So I'm just not sure that, that, that, that would be accurate because kind of, you know, transit and that created a lot of, kind of tensions with their, their fine. The finance department of that particular business because she were very, very skeptical of any kind of modeling as a result coming off that. Fair enough. So I mean Those are two, two pretty bad horror stories. I think the other thing as well is like I heard the other day, huge organization paying millions, millions into an NAN program and they were getting a quarterly report six months after the fact. And you know, I don't know what the point of getting data six months after the fact is. Like, I don't know why you bother. You know, I want data, you know, within 60, 60 to 90 days max, so I can at least use it to draw some inference on my current situation. But, you know, by the time that report kind of disseminates and processes through the organization, you're eight months on from the data. Like, you're, you're way past the next plat, you're way past your planning cycles. And so I think that's, you know, that to me is, is pretty crazy. I've been pretty shocked by how common that is in the States, and I believe it's because nobody in the States has no developed a product like Data OS to solve the data ingestion problem. Because I think that is the, that is the key blocker for most enterprises that I see. The more complex that they get, the more harder it gets to orchestrate data. And the MMM platforms have not been good at building data orchestration platforms to make it easy to get data into their systems.

B (27:21)

Right. And then you just end up explaining the past instead of optimizing future opportunities.

A (27:29)

Exactly. I think, look, and I think there's probably some value in defending the past, right? Like, you know, to some degree, but it, to some degree, it always means we'll be a political tool, not a practical tool. And, you know, I would much rather be a tool of practicality than a tool of politics. I know that wouldn't be the preference of every organization, but I think that certainly the ideology that we bring to an organization.

B (27:56)

It'S a mission I can support. So we're going to take a quick break and when we're back, we're going to talk more about what you guys do with a focus on AI. And we'll also get a little more into the weeds on some M and M stuff, so stick with us.

A (28:20)

Foreign.

B (28:27)

I'm Sarah Sluice, editorial director at Ad Exchanger, and I have with me here today Ludo Develant, the product marketing lead at Amazon Ads, our podcast sponsor this month. Hello, Ludo.

C (28:38)

Yes, hello. Thanks for having me.

B (28:40)

So to start things off, what is the biggest opportunity right now for advertisers in the streaming TV market?

C (28:48)

Well, the biggest opportunity in my view, is to remove the Guesswork for marketers. When you think about it, seeming TV combines the best of both worlds. It's mass reach with precision and personalization. And with Amazon ads, advertisers can achieve this personalization at scale by serving ads to specific audience based on viewer behavior while delivering broad reach. And this powerful combination helps maximize advertising impact and remove more importantly wasted ad spend. And this is really critical because you know, the ANA has estimated that marketer on average waste 36% of their budget through inefficient targeting, duplicative ad delivery, over reliance on probabilistic audiences.

B (29:28)

So streaming delivers that same mass reach people love with TV advertising. But less waste, more personalization. When advertisers consolidate their streaming TV investment with Amazon ads, what happens?

C (29:43)

Well, I think there are two advantages to work with Amazon ads. I mean first of all, Amazon ad is the only DSP that has all premium streaming inventory under one roof. So of course we have prime video ads which is our own property. But advertiser also have access now to all premium publishers including Netflix, Disney, Roku and more. And the second advantage is that we power our advertising solutions through the Amazon Eyes authenticated graph. This is a unique graph which is built on verified relationship and not model data. And so in the US we can reach 90% of household to help advertiser manage through unduplicated reach and frequency and it's delivering great performance. So for instance, with the same budget, advertiser can see on average 42% improvement in unique reach for their campaign with a reduction of 27% of frequency.

B (30:33)

So we've got inventory and identity as the two unique pieces. So let's close with looking ahead. Where do you see advertising on streaming TV heading in the next few years?

C (30:45)

So I think streaming TV is really democratizing access to TV advertising. The barrier to entry are coming down with more self service options without minimum budget or year long commitment. So for instance at Amazon we have Sponsored TV which is our self service streaming TV solution for businesses of any size with without any commitment in terms of minimum budget. And the second driver is AI tools that make video advertising creation both accessible and also very affordable. And I think this means that small businesses who could never afford TV before will join the game. And all in all, I think we could go from thousands of advertisers to potentially millions of TV advertisers in the next few years which will unleash a new golden era for creativity and with more choice and more entertainment for consumer.

B (31:33)

So we will be seeing more small advertisers entering the TV market using AI to create their ads. Totally agree with that prediction. Thank you, Ludo. And thank you to Amazon Ads for supporting our podcasts.

C (31:46)

Thank you for having me.

B (32:02)

All right, we're back. And before we start nerding out again on all things measurement, what the heck does the name Mutinex mean? I need to know.

A (32:15)

So there's a bit of a story around this. We originally called the business Mutiny.

B (32:21)

Okay.

A (32:22)

Mostly because we thought it would annoy our former employer and so we thought it would be quite funny. So we were leaving and we had, you know, we had left our former on pretty good terms, actually. You know, they're pretty good to us, wpp. But of course we thought it was a bit of a, you know, slightly childish, churlish humor, I suppose. But, but so we called the business Mutiny and of course we were coming to America, we saw there was a Sequoia backed company called Mutiny and we're in a VC backed company. It's not a good idea to irritate Sequoia. So we kind of decided to change the name. We, we changed the name to mutinext as a kind of Mutiny Next felt like our next era coming to America. So we kind of just, just did it that way. Little did we know there was a very well known cough medicine here called Mutant X.

B (33:26)

Yes. Yes, there is. So can I tell you something funny? I did a Google search for Mutinex and then under people also ask, under that header there was the question, what does Mutinex with a T do? And then I guess Google got confused because this was the answer. Mutinex works by thinning and loosening mucus, making it easier to clear from the head, throat and lungs. But honestly, it feels potentially like a pretty good reference. Right, because that's what you're trying to do for marketers and finance.

A (33:58)

I see. I feel like marketers in the modern world, you know, facing an increasingly turbulent time, probably have quite a few headaches to deal with.

B (34:10)

That's right. Clear their lungs.

A (34:13)

To some degree. You can think of me as medicine for the market. So. So. So, yeah, so. So I don't actually hate it anymore. I have, I'm great friends with. With someone who I think you. He knows bloody everybody but lupus. Scarless.

B (34:31)

And I do. I do.

A (34:33)

Yeah. And so. And so Lou, he's a delightful human being. He probably won't like me saying that to you publicly, but. But we love you, Lou. And, and so Lou Lou said, hen did you realize that kind of, you know, this is what this is kind of what you linked to. You know, you kind of sound like a. I'm like, no, but I can see that it could be quite funny, you know, So I just kind of think we're going to roll with it now.

B (35:07)

I mean, ask your doctor about Mutinex.

A (35:10)

Yeah, exactly, exactly. You know, or ask your CFO as it were.

B (35:15)

Ask your CFO about Mutinex.

A (35:17)

Yeah, yeah, you ask your doctor about Mutinex. You ask your CFO about Mutinex.

B (35:24)

Well, thank you for clearing that up because I was really curious and I want to talk a little bit about AI, which I teased before the break. Break. I might be contractually obligated to talk about AI on this podcast at this point, but you guys have a feature called mate, spelled M A I T E, which I.

A (35:43)

Marketing Analytics, Insights and Trend Expert.

B (35:47)

Oh, okay. So there's like a twofer on the wordplay because I thought it was just like kind of a portmanteau ish thing where you put AI into the word mate.

A (35:57)

But that was the point. We backward engineered the acronym.

B (36:02)

I mean it's a little ham fisted but it works actually, it makes sense. So it's an AI powered chat based tool that you can interact with and get instant answers to questions about MMM with insights, reports, recommendations, instant scenario planning, I'm sure other things, all that jazz. But talk to me a little bit about that, make it real for me with an example and then zoom out if you could and talk about AI assisted mmm, because that is a trend. Now people are talking about it. I don't know how real it is.

A (36:38)

Well, so let's start with how we use it. So again, I think the most important thing for MMMs and most market Mitch modeling programs are surrounded by a vast army of consultants who are putting together hundreds of slide decks in order to respond to executive questions. I think this allows you to provide a far more tailored experience to directly answer the question a user is asking, which reduces the cost to educate them on the 50 platform features that they would need to get that answer. So I think practically that's what it looks like for us. Um, you know, I know of. There's, there's like, there's. I wonder if I can share. No, I can't share screen here, can I?

B (37:32)

But no, this is a podcast.

A (37:34)

No, but, but I was really practical example. In fact I'm just going to just like bring something up just because I will give you some examples of questions that are being asked, but I'm just going to look into amplitude and just read a couple because it's just interesting.

B (37:51)

Yeah, do it.

A (37:52)

So we have amplitude running. Amplitude is obviously our kind of core infrastructure here to kind of track what's happening in our system. So I'm logging onto the mate dash just to have a look. So in the past, kind of past couple of days we've seen about 170 queries successfully answered. And you know, people go through different average kind of usage pattern that people don't ask around, you know, one to two queries in a session. But the queries are often quite detailed. So here's a really good example. Was there any brand variance within FTA tv? I'm. Which underperforming channels could we reduce to free up budget for testing? You know, what's your, what's the MRI impact across social between different format types? Can you, can you look at the product volumes for the last three quarters and tell me what's important? Can you please look at our brand baseline and how it's been growing in recent times and what drives the growth in it? Can you explain baseline sales drops over time? So it just gives you a kind of a sense of it. Like it's quite detailed questions and quite specific questions and I think they would, you know, the typical process that you would take before this kind of technology existed is you would, you would really speak to an analyst who would probably brief a team of two to three other people and you would be going to kind of compile all of that information together and then respond back probably for a slide presentation in a 60 minute meeting. And that means that the cost of answering that question is very, very high and the internal cost of ownership of the MMM is very high. And so I take a fundamental view that what we want to see is that internal cost of ownership of MMM plummeting through the use of AI. And so that's what we're doing. The second place that we're using it is we've gotten very, very good at helping customers organize their data using AI and automatically build data warehouses. That's something we've gotten very good at at doing. And I think that's been a huge value add because the conventional assumption is that you need structured data for a market mix modeling program. You absolutely don't anymore. So, you know, we don't need structured data, we just need data connected to data OS and then the agents will build the warehouse for us. So I think that's a really, really important component to it. Zooming out to go to AI assisted mmm. Well, I think there's three things I'm seeing one is the Insight packaging which is kind of where. Where I think we're probably the market leader in that. I think we will be within a few weeks because we have a large product announcement coming that will certainly solidify us as having the best marketing, marketing effectiveness agent on the planet.

B (41:18)

Share it with us, please.

A (41:19)

I certainly will. It will be a very large scale content deal. So which will mean that we'll have access to effectiveness content that no one else on the planet has access to to fuel these agents and give them context. But so I think that that Insight packaging is one area. I don't see many people doing that. Well I think there are reasons for that because I know where our technical stack diverges and so a lot of what we've done done earlier in the stack means that it's more effective to be. We quite effectively do that and deal with quite complex queries without it breaking our system. So I think that's important. The second area that I'm seeing that we're not in is synthetic audiences. So people kind of building. Building synthetic responses to fill in some of the gaps in MMM and kind of using the LLMs in that space. I think it's working for now from what I can see. Meet next is making a conscious decision not to invest in that area mostly because I have a long term hypothesis on the future of synthetics which is I think that more and more AI content is going to be generated and curation of data in curation of training data into the foundational models is going to become more and more fragmented and I think more and more publishers are going to gate their content to force the AI companies to buy it, as they should. I think it's outrageous that the AI companies have gotten away with breaching hundreds of years of copyright, which is a completely different issue, but I think it's completely wrong. And word. And so with the, with the publishers gating content I think you'll see different foundational models have access to different training data sets which means it will become much harder for them to synthetically represent the entire population because that premise relies on you accessing the data of everybody to be able to synthetically represent everybody. If you work off the assumption that all the data is going to fragment or just look at what Elon Musk is doing and saying about training. Grok right. He's actively saying that certain voices are. He's going to train the models in certain ways to go for maximum truth seeking. But there's a whole portion of the planet that probably lies or at least in his worldview, would lie. And so as a result of that, does that mean that they're getting excluded and therefore synthetics coming out of GRO would now no longer be as reliable. So I think there's like a bubbling problem for me there in synthetics that doesn't exist in the early stage of models, but may exist in five years. And so that's where I kind of worry about synthetics degrading in quality long, long term. And so that's an area I think I see a lot of activity in. The third area is can we use AI to build MMMs from scratch and things like that? The answer is no, not from what I can see, because they can't solve the obvious indigenity problems, they can't solve the obvious causal inference problems that exist in mmm. And you know, because half the battle with MMM is not. It's not, you know, it's knowing, having the domain knowledge to apply in a very specific circumstance to understand how to construct and apply that model and what inputs to collect and all of those sorts of things and how to, how those inputs have relationships to each other. And so I think that's a hard problem to solve. In the current state of LLMs, I do think that there is a likelihood that LLMs could subsume the sector and then the infrastructure around the MMM becomes more important than the construction of it itself. But I don't think we're there yet. As to where I see players playing in space, I don't really see a huge amount of AI in the MMM space other than Insight packaging at the moment. That's not to say I don't think it won't exist. I just, I don't, I don't see it as being too high priority. I think there is a reason why, because most people in the MMM space come from data science backgrounds. And so I think that we are naturally more skeptical and anxious about the risk of hallucinations. So most of the people designing the products probably have a higher retinance to introduce products with a hallucination rate of some kind. I have a slightly different view on hallucination rates, which is, I think for me, hallucinations are effectively. How often does the AI just make something up or bullshit about something? In my mind, a hallucination rate of 4% is probably, if I look at a human, if I'm talking with a human, what percentage of their conversation is exaggerated or made up? So most humans would probably be higher than 4%. So I'd say AI hallucination rates in large language models. They remind me a little bit like self driving cars, like self driving cars per mile basis are safer than human driven cars. I think LLMs lie less and make stuff up less than humans en masse. So therefore I'm not necessarily sure I see it as big of a problem.

B (47:23)

I'm thinking of myself at a cocktail party or something where you get forced into a conversation you don't want to be in. And the stuff that comes out of my mouth, I mean mostly nonsense. I'm just trying to extricate myself, but I don't even know what I'm saying.

A (47:37)

Exactly. Exactly. I think the great irony of hallucinations and how we talk about hallucinations and AI is we even that word hallucination is a kind of slightly funny one because really like we're talking about making something up and it's a very human trait to do that. And so you know, when you train something off the entire corpus of human insight, memory and things like that, that's what you accept the Internet to largely be. You know, I wasn't, I'm not really shocked that, that you know, it's learned to make stuff up in certain situations to sound more confident because I think that is what most humans do. I think it's just adopted a human behavior rather than adopt, rather than it being a bug. I'm not, I'm not sure it's a bug. I think it's a feature or a reflection. Yeah.

B (48:35)

And to go back to the very detailed questions that you're finding your clients asking of mate m a I T E, what are the answers? Like if someone's asking some very specific question, they obviously want a very specific answer. So rattle off one of those questions again and like what would the answer be that the AI could provide?

A (49:00)

It'll be very detailed. It would be able to, it would be able to kind of bring in, it would be able to bring in kind of, you know, quite detailed information and things like that. Sorry, my, that might just, Sorry for my partners just ordered some room service.

B (49:19)

So this is, this is, this is real, this is real life. Yeah, sometimes partners order room service.

A (49:25)

So, so, so, yeah, so I, I think, you know, the, the, the answers are quite detailed. Generally speaking, they're very data backed. We have, we've spent a lot of time trying to train the AI to respond with detail and data and in the way that we would expect a senior analyst to respond. And that has been a huge amount of work over a long period of time to supply it with enough responses and context and structure to be able to do that. And so, I mean, it took us 18 months to really get to that point. And, you know, we invested a lot very early on trying to do things in, you know, we tried retrieval networks at the start. You know, we have, we've had to build some pretty complex systems to do it and to train these agents to behave the way that we want them to. Now we have it to a point.

B (50:32)

Where.

A (50:34)

It does seem to perform very well. The one drawback of that system, in order to get to that detailed response and to do it accurately, we have a. And to do it with no kind of mistakes being made on the data being supplied and things like that, we have had to build a system which probably takes more latency than we would want. So if you think about, you know, a chat GPT query taking maybe 30 to 60 seconds to resolve, I'd say ours are at about 90 seconds, which can often feel a little bit clunky in a user experience. And so. But, you know, I often compare the 90 seconds and you know, I was kind of having a whinge about it to a client the other day and they said, well, yeah, it is 90 seconds, but you know, previously it was eight weeks for my team, so I'm not that unhappy with it.

B (51:25)

Yeah.

A (51:26)

And so, and so I thought, yeah, you know, I thought that was a pretty fair comment to make. But having said that, you know, I always think the iPhone, I always say the iPhone moment for our product is, you know, when a CMO is sitting in that room with the CFO or someone like that and, and you know, rather than saying the six words that lose them control every single time, which is. Let me get back to you. Because when you say those words, you lose control. Whereas the moment we know we've done a brilliant job, you know, forgetting all of our product metrics and things like that. I know the magic moment we want to work towards is when a, A c, A CMO sitting in that room and they go, and they type that question in as it's being asked and they can answer it there and then in the room and they can answer it brilliantly and it puts him back in control. And I think that that's really what we're bringing to the table. That's why I think it goes beyond models. You know, we're really upgrading the modeling infrastructure into answers infrastructure. And I think that's a really important shift because that's how we're going to change this sector from, you know, the measurement sector is kind of a weird hybrid of, you know, data nerds with pretty dashboards and you know, and a lot of consulting manpower. It's not truly a product category as of yet. It's not a category that anybody would claim has particularly high NPS and it's not a category that anybody would claim. You know people fall in love with it, in love with the products, you know, they can't live without them. They kind of tolerated as necessary evils. And so I think that's, you know, what we know once we get to that moment. It's a very, very good one. A great story is I have a client that very early on in the mate journey I saw the mate queries they'd spike every Monday from 22 to 4pm and I, and, and I called them because it was quite early on and I said, I said why, why the query spiking at this, this, this time now, what's, what's going on? They said oh, we sit in our executive team meeting, we've got mate weapon and, and rather than kind of go and run around and ask all the teams everything, we just ask it the questions we want to ask about growth it there and then in the meeting and it's like having all the analysts in the room and, and I thought that was, that was probably the moment I got conviction that this would be a, this could be a multi billion dollar product.

B (54:13)

Well, are there ever marketers you work with who just refuse to see what's in front of their eyes? A lot of marketers have certain ingrained notions about like what channels they should be in and if they want to run a big splashy TV campaign and you come and tell them it would be way more eff to do something else that would perform better. Like I'd imagine they might ignore the data and just go with their gut. So yeah, how do you, I don't know, like how do you handle that kind of resistance to data driven recommendations or do you not get that kind of resistance?

A (54:48)

Well, I think that really comes down to how, how you present these platforms. Right. Like I wouldn't necessarily say that you want, you want to go and try to reallocate everything all, all at once. You don't kind of go from 0 to 100 straight away. You know, we, I wish it is a bit of a hierarchy, prove that you can make a change within a channel. So you know, we look at creative ROI and format roi, publisher ROI as well within a channel. So we're just quite unique. Once you've proven within a channel, then Prove within channel between channels. Once you've done between channels, then prove between geographies. Once you've done up between geographies, then getting go between large product lines or large product priorities or, or, or even brands within, within a kind of house of brand structure. So, so I think there's kind of a hierarchy of proof, if that makes sense, that you kind of work things along. And I think once you're doing that, that's how you build organizational confidence in these areas. I don't really think people should take a model on face value. I think that they should be skeptical and try to validate how the model performs in the real world before they make big strategic bets. And that's the culture of our organization is to kind of, of run through those testing frameworks. And so, and I think that's why we're able to kind of get rid of that skepticism because we, we, we built the pathway in my mind to solving that problem.

B (56:34)

And so we're, we're nearly out of time. So I have a last question. If you could ask every CMO in the world to run a single marketing measurement experiment, what would it be? I'm basically asking you what you want for Christmas. Like it's not a pony, but you know, still.

A (57:00)

I'm not sure it's an experiment. Might be the wrong word. I've always felt that trademarks should defend branded should at branded search, that you shouldn't have to pay for your own to defend your own brand to show up in search. I think that's like a slightly, slightly challenging thing that exists across the search engines. And I wonder what would happen if we all turned off both competitor and branded search, not so much from the experimentation on an individual brand level, but just from a ethics of advertising and trademark kind of position. And whether that would create a healthier ecosystem, which I think it would, and whether that would put a lot more margin back into growth. I would, I'd be very interested for every CMO to run that experiment. I think it would be of an incredibly brave and interesting category to go if a category got together and all the CMOs agreed, let's stop competitor and branded bidding so that we're only investing in things that grow our category rather than things that, rather than compete against each other. And I wonder if that would deliver superior category growth for the category that decided to take that kind of leadership position. It's a bit altruistic and optimistic, but I think it's an interesting idea.

B (58:32)

Certainly everyone would have to do it, but it would be really, the results would be very interesting.

A (58:38)

Yeah. I mean, you know, there's enough categories that are small enough and localized enough that, you know, all the telcos in one particular country can certainly probably do that.

B (58:49)

And the telcos hate working together and.

A (58:51)

Come now, but they might grow their categories. Sure. So. So, you know, I don't know. It'd be, it'd be an interesting experiment, I think.

B (59:01)

Well, I. I'm gonna let you go because for our listeners, just so they know, it's now about 8am in New York where I am, and it's about 11pm where Henry is, and his partner did just order room service. So I think it's time for him to go share in that.

A (59:17)

Yeah, Allison, it was awesome.

B (59:29)

This episode is sponsored by Amazon Ads. Amazon Ads offers a range of products and solutions that can help businesses achieve their advertising goals. Advertising needs a world where marketers no longer have to choose between building their brand and driving results. Amazon Ads helps marketers prioritize solutions that break down silos and simplify campaign management, enabling the orchestration of execution and measurement of holistic campaigns that achieve both objectives. We remove the guesswork for advertisers by making it simple to manage all of their TV planning and buying with Amazon Ads.