← AdExchanger
AdExchanger
The Behavioral Economist’s POV On Marketing Measurement
00:49:51
Marketing measurement is messed up – it’s hard to argue otherwise – and most people agree on the reasons why: vanity metrics, imperfect models, unrealistic expectations. But there’s another issue at play, argues Julian Runge, an...
Loading summary
Transcript78 lines
- [00:00]
Julian Runge
Foreign welcome to Ad Exchanger Talks, the.
- [00:15]
Alison Schiff
Podcast devoted to examining the issues and trends in advertising and marketing technology that matter most to you.
- [00:24]
Sarah Sluice
This episode is brought to you by Mutinex. Pioneers in AI Powered market mix modeling get fast answers to hard questions with Mutinex, your growth copilot. Ask for a demo at Mutinex Co. That's MU T I N E X Co.
- [00:52]
Alison Schiff
This is Alison Schiff and you're listening to Ad Exchanger Talks. So let's nerd out. If you love advertising measurement, then this is the episode for you. My guest this week is Julian Runge, Assistant professor of Marketing at Northwestern University in the Medill School of Journalism, Media and Integrated Marketing Communications. We'll talk about why marketers don't embrace true experimentation when it comes to measuring their marketing, and why they really, really should start. We'll also get into an area of particular passion for Julian, which is the fact that gaming still isn't getting its due as a valuable and highly performant channel. But first, please allow me to make a quick plug for our next conference, Programmatic IO Innovate, which is just around the corner, metaphorically speaking. It's actually in Las Vegas from May 19th through the 21st. We've got a great agenda lined up for you, including highly practical sessions on attribution. Bad news? You're probably doing it wrong. A fireside chat with Paul Longo, General Manager of AI in Ads at Microsoft Advertising, a workshop on how to move beyond the buzzwords of AI and into meaningful AI integration, and lots, lots more. We'll also be recording a live episode of the Big Story podcast, which is always fun, so visit our website to learn more and reserve your spot. Podcast listeners get 10% off as a thank you for listening. Use code POD10 to get your discount. Julian, welcome to the podcast.
- [02:21]
Julian Runge
Glad to be here.
- [02:22]
Alison Schiff
All right, what is something about you like a fun or weird or just noteworthy fact that not a lot of other people already know and that I couldn't find out just by checking out your LinkedIn page or giving you a little Google.
- [02:39]
Julian Runge
Yeah, so not sure it's noteworthy, but I love water sports so I surf, I have wakeboard board, I've kite surfed and whenever I'm not by a big body of water where I can do that, I walk a lot. Like I think my average step count for this year as counted on my phone so the real number is probably higher is like around 13,000 steps a day. So yeah, I definitely walk a lot.
- [03:08]
Alison Schiff
That's impressive. I thought that I walked a lot And I was averaging something like seven or 8,000 steps, steps A day. And then I looked it up and apparently you're supposed to walk 10,000 steps a day, like that's what's recommended. And I was really annoyed. I'm like, I thought I was walking a lot and I wasn't even hitting what I'm supposed to be doing as a baseline.
- [03:30]
Julian Runge
So yeah, I think the 10,000 is a little arbitrary though. I wouldn't, you know, I think any step counts kind of. And for me it's actually a habit that developed. It's like this microhabit thing. It actually started with a pandemic because I was like, okay, really hard to do any sort of activities like surf or play basketball, which I also used to do. And so I started walking and I think it's been going up every year by around thousand steps per day extra. And Now I'm at 13,000. So it's really like this microhabit that just kept growing.
- [04:05]
Alison Schiff
So I'm going to make a little segue between counting your steps and measuring advertising. So it's a little ham fisted but it kind of works. But first though, I, I want to talk about what you're, you're doing now. You're an assistant professor of marketing at Northwestern University just outside of Chicago. You're doing research, you're teaching quantitative marketing and applied data science. And you've been doing that since September. But a little bit more background on you for our listeners who might not know who you are. You were going way back. You were head of analytics and data science at Wooga for five years, German Gaming Studio between 2009 and 2014. And then it looks like you transitioned into academia and research. You were at Duke for a time, but you also spent some time at network gaming publisher that bought Eric Suefert's outfit a while ago. And the the IAN network is a backwards, it's like a backwards E because it's a number three. And you were a research scientist focusing on marketing science R and D at Facebook for a few years. 2019, 2020. So on LinkedIn you describe yourself as a behavioral economist and quantitative marketing professor researching the intersection of US algorithms and immersive media, which is very cool. Also a mouthful. What does that mean exactly?
- [05:35]
Julian Runge
Wow, this is some solid background you got there. And just one note, it's UX algorithms.
- [05:42]
Alison Schiff
What did I say?
- [05:43]
Julian Runge
I think you said us, but maybe I misheard.
- [05:46]
Alison Schiff
Only algorithms in the U.S. yeah.
- [05:49]
Julian Runge
So user experience algorithms. Yeah, I'm really fascinated by those and love working with Them. Yeah, that is some, as I said, super solid background on me just in terms of timeline, maybe. Yeah, I was at Vuga before I did my PhD and then during my PhD I went to Stanford University, invited by Jonathan Levov, who's a professor there, and we started working together and that kind of. I came back the year after 2017 and then I never, never really left. And I really like working with field experiments that I conduct with companies to evaluate algorithms or different marketing treatments in the field. I think experiments are just great because they really allow you, at least when they have a control group, they allow you to make this statement on the treatment effect in the most rigorous and unbiased way. Yeah, and so that's really me. And then after the PhD, as you said, I went to Facebook at the time, joined their advertising research team, working on academic collaborations and writing white papers about the measurement platform, actually specifically focusing on experimentation and how companies use experimentation on Facebook to measure the advertising and what impacts that has on their economic well being and performance of their advertising. And then from there went to Duke for a year as a visiting scholar, invited by Carl Miller, who's a big quantitative marketing professor there, and then got a two year appointment at Northeastern University in Boston at the Mora McKim Business School and from there went to Northwestern. Recently, as you said, important to note, I'm not at Kellogg, that's the business school here. I'm at Medill, that is the school of Journalism, Media and Integrated Marketing Communications. And I think what's really interesting there is like you may wonder, why does Northwestern have two marketing departments? Kind of the history for that is that advertising, I mean is and used to be really big for news. Right. And for journalism as a way of monetizing content. And so there was this advertising group inside Medill and that is kind of what grew into what is now called Integrated Marketing Communications, which is the department that I'm an assistant professor in.
- [08:06]
Alison Schiff
I mean advertising and marketing is still pretty important as a revenue driver for news publishers is just really, really hard to do and getting harder. But I wanted to go back and I guess re ask my question, like if you are a behavioral economist focusing ux, not us algorithms, like what do you, what do you do with your day? What are you doing?
- [08:33]
Julian Runge
That's a great, great way, a great question. So I, I believe in experiential learning and also in my research I try to embrace that approach to the extent that I actually play games, you know, especially mobile games, which I think have offered this fascinating new way in terms of how games are being played by people and then think through ways that, you know, what are the meaningful levers that actually shape those experiences and allow us to learn something that generalizes beyond the specific game. And clearly in marketing, very important is price and promotion, right? So price, personalization, targeted offers and the likes are very, very important tools to drive revenue in freemium apps and free to play games. But it goes deeper because the fascinating thing with games or immersive media to me is the degrees of freedom we have to shape the user experiences. So when we think about further important components, there's social matching. In many games, players initially play the game, maybe go through a tutorial, and then we'll join a group of players where they play together either to beat virtual opponents or also to, you know, collaborate or compete amongst each other. And there's so much that can be done by using machine learning to decide, like what teams or what groups do we now offer to what players and what impacts does that have on, on their social behavior and on the engagement of these groups. And then of course, also on monetization in a second order effect. Another thing is dynamic difficulty adaptation or just like on the content dimension of games. One of the key things, of course, is how difficult is this game now? And you can just do really amazing things if you make sure that the difficulty of the level that a player now plays or has to beat really is tailored to their skill level and motivation level in that moment. That can just create so much more delight than if you don't use these tools and don't use algorithms to tailor these experiences. So this is the stuff I'm really fascinated about. And you know, I'd say like marketing as a field, some of this stuff is relatively new still for many marketing scholars, because gaming hasn't necessarily traditionally been a context that wasn't researched as much. And to be fair, mobile gaming has really also just increased the number of eyeballs that are bound in gaming massively. I think we now have more than 3 billion digital gamers in the world and that's only possible because of smartphones. And so, yeah, this is really what I'm fascinated by and what, what my work focuses on.
- [11:17]
Alison Schiff
Well, and yet let's talk about gaming. I mean, marketers somehow still kind of ignore billions of people. The fact that billions of people are playing games. I mean, if anyone attends the IB playfronts, they'll hear the same sentiment expressed year after year. You could almost attend as a journalist and write the same story and just change the date lines. And who's saying the thing, but basically it's just bafflement from all of these ad tech vendors and also agency people and other folks that are really confused as to why marketers aren't embracing gaming as marketing media. Even though so many people are playing games. Like, I mean, a large portion of the world, the planet's population. And I mean, if you think your audience doesn't play games or it's not a good way to reach people, then you're nuts. I mean, one third of the planet's population, right, something like that plays games. So for the love of all that's like, I don't know. Holy. What is holding marketers back from treating games like they treat other channels as a way to reach their audience?
- [12:26]
Julian Runge
Yeah, great question. So that's something I've been thinking about a lot recently, and I think there's a few things. So one thing is definitely that there's still misconceptions or preconceptions about gaming that just don't apply anymore. Right. Some people still think about gamers as like, you know, young kids locked in their basement playing for 24 hours in a row. And that's just not the predominant model that games are played anymore. Right. With games like wordle or Candy Crush and highly casual games that are exclusively often played on mobile devices, that just doesn't apply anymore. And by virtue of that, games now reach all kinds of audiences across age brackets, across income brackets, across different other demographic or socioeconomic variables that are commonly important to define audiences. And so I think it's just a process of that also trickling through. Right. Like, because this whole shift has started with mobile phones or smartphones becoming so prevalent and still, you know, that that sector of the market is still growing right now, I think about 2/3 or almost 2/3 of revenues in gaming come from mobile phones. And yeah, as we said, like, more than 3 billion people play, I think. And so just like if we had a social network with 3 billion people on it, nobody would dare to not think of that as one of the most important advertising channels possible that exists.
- [13:53]
Alison Schiff
Exactly.
- [13:54]
Julian Runge
Yeah. So, I mean, I will grant that there are differences, of course, between games and social media. Games tend to be a little more immersive. It can be harder to place advertising in a smart way, but it's possible. And I think we're currently not really leveraging or the market isn't really leveraging this in a way that would be possible. I think some leading brands are definitely diving into it. Walmart, for example, I've heard, is experimenting with in game banner ads and things of the sort. We have a lot of branded experiences happening, especially on social platformers like Roblox or Fortnite. Lego's partnership with Epic Games and Lego Fortnite was hugely successful. I think that pulled like 2.4 million players into the Lego experience on day one and in the quarter after, they had a 26% surge in profit, I believe. And so these are some ways how games are being leveraged by leading brands, but I think there's just so much potential it's still being not leveraged at the moment.
- [14:55]
Alison Schiff
And what's the unlock there? Is it just that marketers need to change their perception of gamers? Because that, that's been true for a long time. I mean, you could also, I don't. You can go back five, six, seven years and people have made that observation, right? Like these aren't kids wearing hoodies sitting in their basement. So is it a measurement thing? Is it proof? Do they need more proof?
- [15:20]
Julian Runge
Yeah, great questions. I think one of the, maybe one of the root causes is actually fragmentation, right? So we discussed the inertia, the preconceptions. That's all true too. But gaming is heavily fragmented in the sense that you have like hundreds and thousands of studios who make hundreds and thousands of games that are then being played by roughly 3 billion people. And in social media, that is different. You have like a handful of platforms or companies that really dominate social media. And then of course, when one company manages such a large swath of consumer attention, if you want to call it that, it's much easier to, as you said, build the appropriate measurement frameworks, build a good targeting infrastructure, and do all these things that are needed for an advertising medium really to find its place inside mainstream marketing operations. I think that's one thing that's just inherently tricky. But to say this, we do see consolidation in gaming, meaning that we have a few publishers who are actively acquiring additional company or other companies to grow, basically how many players they manage, just to name a few. Those would be take two scopely, Tencent, Microsoft and Activision Blizzard. And I think these companies will be in a position to really produce a shift there over time. One thing then is also of course, just consistent ad formats, right? Like even getting it across to brand marketers what they can do in games, ranging from product placement to in game billboards, to in game goods that are branded to banner ads, et cetera. And just having some consistency. One thing I want to call out in that regard is a recent initiative by Roblox, just Like a few weeks ago where they worked with Google, Nielsen, Cantor and DoubleVerify to kind of offer a standardized ad format in end measurement, coming with that in conjunction with these collaborating companies. And I think that's really exciting stuff. And as we see this, I think we're slowly, the market is embarking on actually producing what's needed for games to become a more mainstream marketing medium.
- [17:34]
Alison Schiff
Well, you're obviously very passionate about this, and I know you're also very passionate about experimentation. And you've written about that for Ad Exchanger with Igor Skogen, the Global Marketing Science director at Meta, and also Bill Grosso, the CEO of Game Data Pros. You wrote this twin set of articles. I think the headlines really say it all. So part one is the uncomfortable truth about advertising effectiveness, why marketers avoid true experimentation. And then the second piece is about how organizations can embrace experimentation and marketing measurement. But yeah, I mean, the uncomfortable truth about advertising effectiveness is that marketers avoid true experimentation. That feels like a real condemnation. It feels kind of crazy at this point that marketers are often so hesitant to embrace something that will tell them the truth about how they're spending their ad dollars. And despite all the evidence that if they don't do it, they're probably misallocating their resources, they still don't do it. So why don't they do it? Why won't they do it?
- [18:44]
Julian Runge
Well, I mean, actually it's a similar story. I think they are starting to do it. I just saw on LinkedIn a few, like Marketing Science Institute just ran a few surveys among different enterprises of different sizes. And one could actually note that larger enterprises have started embracing experimentation quite a bit more and it's now starting to rank similarly in importance to marketing mix models. So I think there also, the market is probably listening to some of this stuff, others. And also I have been writing over the last years and yeah, so I think this is, this is slowly trickling in. I think the, the, the problems are quite, quite similar. Right, because measurement is only very, I mean, one part of the equation that generates successful marketing where good strategy, good branding, messaging, creative, etc. Are also crucial components that I guess often are easier to control. Right, because experiments often require you to work with other platforms, to work with specialized vendors, etc. To actually get that done the right way. And it's very common in management generally that people like to like, prefer to focus on what is actually in their control, which is often the messaging, the branding, the creative, etc.
- [20:08]
Alison Schiff
Well, let me ask you, if there's not A weirdly emotional component. Like some marketers don't do the hard work of true marketing measurement experimentation because it's hard and it's time consuming and all of that, although it's changing. But also maybe because they don't necessarily want to discover that they've been spending inefficient. Inefficiently. Right. Like denial of proof that they haven't been doing their job as well as they should. Or they could. And then maybe when they find that out, they get less budget the following year if they share those results and no marketer wants to get less budget.
- [20:47]
Julian Runge
Yeah, that. That is true. And I think, I mean, this goes back also in some ways to organizational complexities or also different levels of organizations where, for example, the marketing budget is set at the top, like C level. And then you have several tactical level, several operational levels down to the tactical level, where people actually implement stuff with different media channels. And, you know, the knowledge that a channel is doing really well might already be present in the tactical levels, but it just takes time until that knowledge is also making its way up to the senior levels, where the big decisions on how many dollars go to what medium or to digital versus offline, et cetera, where these decisions are being made. And additionally, I think it's often the case that like traditional measurement solutions, like marketing mix models, for example, will be maintained by central analytics teams that are in a close dialogue with the strategic levels. So the executive levels that set those budgets, but are not necessarily in close touch with the tactical levels. So again, there's like a missing link, maybe a little bit getting some of this knowledge that the people who are really in the market, working with specific media have for that to make its way up. So inertia, I think, is actually a big component there.
- [22:10]
Alison Schiff
I mean, inertia is a big factor in my own life too. So I get it. But we're going to take a quick break and when we're back, I'm glad you brought up. Mmm. Because we'll talk a little bit more about that. We'll talk about incrementality and we'll also talk about the how, like, how can marketers that really want to embrace true experimentation do it? Because it's one thing to decide to do it and it's another thing to do it. So stick with us.
- [22:48]
Sarah Sluice
Hello, I'm Sarah Sluice, executive editor of Ad Exchanger, and I'm here with Henry Innes, the co founder and global CEO of Mutinex, with a focus on driving product development. Henry leads the Mutinex product vision. He's obsessed with what's next for customers with a background in software engineering, marketing strategy, works with a lot of the world's largest brands. Welcome, Henry. Let's dive right in. Every organization talks about being very data driven in their marketing, but a lot of them struggle to get there. What's holding them back, in your opinion?
- [23:24]
Henry Innes
I think the number one problem and issue holding any organization back is the cost of asking a question. So if you think about the cost of an organization asking a question, when a CMO goes to ask that question, what tends to happen is a cascade of things happen. People look at various platforms, they look at various sources of truth, they wrangle that together in a presentation, and it takes weeks and weeks and weeks and arguably cost millions and millions of dollars. So the number one way to solve that is to build systems and processes that go beyond just aggregating and building models around data analytics, but are able to deliver answers to complex questions quickly, immediately, and without cost. And if you can start to solve that problem, that allows you to answer more questions. When you can answer more questions, you can become more data driven.
- [24:13]
Sarah Sluice
So there's a lot of analytics tools out there. Now. Why, why isn't it that the tools are solving the problem?
- [24:21]
Henry Innes
Well, I think if you have a bunch of data scientists building tools for other data scientists, it becomes ridiculously hard to build something workable for actual organizations and users. So I'd say that's probably the first thing. I think ultimately it's because we're all focused on the wrong thing. We're focused on more data, not good answers. And I think ultimately every data product focuses on generating more and more data for organizations, which adds more and more cost to organizations. When you have more and more cost to organizations being added from more data, what ultimately happens is they get overwhelmed. How many times have we seen analytics platforms being added in only for new teams to spring up around them, new owners to spring up around them, and them not to be integrated into the business? So we focus very much on ignoring the data problem. And in fact, you know, we know organizations don't want more data, they want less data crafted to better answers. And so ultimately that's, that's what we shoot for, right?
- [25:22]
Sarah Sluice
Less data crafted to better answers. I really like that as a North Star. How do, how do you fix this answers problem?
- [25:32]
Henry Innes
Well, the first thing is to acknowledge that no one wakes up in the morning wanting to buy a market mix model. Everyone wakes up kind of, you know, going, I have a problem. What's the data I need to understand to answer that question. And so that is the product that you're ultimately trying to build. You're trying to build answers platforms, you're not trying to build market mix modeling platforms. The entire framing of the MMM industry around models betrays the fact that it's not a customer centric or focused industry. And so I think if you start to change that fundamentally and start to go, okay, how do we get really good answers where we have very fast models that we know are very accurate and have extreme granularity so that we can actually select the right components that matter to analyze a problem, rather than providing very high level generic answers with lots of strategic waffle to justify those answers, Then we start to get to a much better place.
- [26:29]
Sarah Sluice
So I hear you alluding to what you're building at Mutinex. How does your team tackle this challenge? Give us a little more detail.
- [26:38]
Henry Innes
So we tackle the challenge firstly by, you know, making data ingestion very easy. You know, if you, if you can't take in unstructured data, if you have to spend weeks and weeks in data, it's impossible to get to a good answer quickly. We then focus on having models that run really fast in a very generalized fashion. The reason why is that means those models are not biased by analysts and they also stand up over time and they're more robustly tested across different environments and scenarios. And that means they're more reliable for forecasting, they're better in parameter recovery, they're better in holdout testing, which are objective tests that you can run. And finally, we then build usable interfaces that sit over those generalized models that select the right answers with the right level of granularity to present the information that a user needs rather than forcing the user to consume lots of data that they don't need.
- [27:34]
Sarah Sluice
Well, I like this approach. It's very thoughtful of what people need versus just trying to maybe push another tool for the sake of a tool. As we were talking about early in the conversation. Thanks for coming on the podcast, Henry, and thank you to mutinext for supporting our podcast.
- [27:53]
Henry Innes
Thanks.
- [28:09]
Alison Schiff
All right, we're back. And yeah, let's, let's talk about the how. I mean, what's some advice for a marketer that wants to embrace experimentation in, in their marketing measurement? How do they get started? Who do they have to convince internally? Do they need to hire people? How much ongoing training do they have to do? I mean, it's, I'm. You won't be able to tell me every single step right now. But you know, they're ready for it. How do they get started?
- [28:40]
Julian Runge
Yeah, I mean, super important question, I don't want to place too big of a plug here, but I actually went on Eric Sofer's podcast, mobile Dev Memo podcast with Kuhn Powells, who's a former colleague at Northwestern University, and we talked through this in some somewhat more detail. There's so many ways to do this right and also ways to not do it right. But you do want to bring in probably people who have experience with experimentation or at least have the statistical know how of how these methods work. Usually you will already have these kind of people in, in your data science team. I think that's important. Then executive endorsement, right? This may be trivial, but like you do want an executive who actually sponsors this initiative. It's unlikely that, like coming back to this example I mentioned earlier, it's unlikely that somebody on the more tactical level is able to actually get across to the higher ups like how effective a medium is. And they may be seeing like, oh my God, we have 2x advertising effectiveness here. But then they tell that to people. Yeah, okay. And then an executive I think is needed to actually work out the measurement plan and put in place like kind of breaking down the hierarchies or silos inside the organization to actually make sure this knowledge then also gets built into the MMM that the central analytics team maintains, et cetera. And I think with that over time you just probably ideally have also a change in organizational culture of more of a culture of failure also where it's okay to run an experiment and it doesn't work out. You also shouldn't call this too early. Right? You often need to rerun experiments because also experiments aren't perfect. There can be interference, there can be dynamic effects over time. And so you want to run several experiments before you really call something. And I think these things can be quite helpful in getting this started. But in the end, I think if you want to make it super concrete, go down to people who actually work with the media, ask them, hey, what do you think? Where do we have the biggest misconception conception in terms of the effectiveness of one of our advertising channels? Right. This could be something digital, I don't know, Snapchat or Instagram. Or it could be something like more exotic like point of care marketing if you're at a healthcare company. And then really work to set up a measurement plan, including several experiments for this medium to really know what it does and see if it aligns with the outputs of the central analytics mmm Right. That's one very concrete way to, to get this started, I think.
- [31:25]
Alison Schiff
What do you think of mmm? Is it good? Is it bad? Is it ugly? Does it only have value if you, if you use it for what you're supposed to use it for?
- [31:36]
Julian Runge
Is it good, bad, Is it ugly? They can definitely be ugly mmms I think. But yeah, overall I think mmms are good. I'm a big fan of measurement generally and also of MMM specifically because they're the tool that allows you for that comprehensive strategic overview of in terms of like how your marketing is doing. Right. They won't deliver you all the answers you need, especially technical answers. They often can't inform because they're more focused on the longer term strategic side of things and you know, actually making sure at a broad level are we doing reasonable and effective things on the marketing mix? And just to call it out this out once, like the marketing mix isn't only advertising, it also includes pricing, promotion, what you do on the product side. And this is actually also where this kind of loops back to what I was telling you earlier about user experience algorithms. So UX algorithms that I'm so fascinated by and, and work on a lot in gaming specifically. Right. Because in the end an MMM can inform like all the different strategic choices that you make as a marketer across these four piece to make sure they're overall kind of well orchestrated. So yeah, I'm a big fan of MMMs for that very reason.
- [32:55]
Alison Schiff
And how does incrementality come into it? So you're running your MMM model and you also want to measure real incrementality. How do you do that? I feel like it's a question I've asked a lot of people and I'd love to hear what you have to say.
- [33:12]
Julian Runge
Yeah, well, I think it goes a little bit back to the articles like the two part op ed we wrote for Ad Exchanger. You do want to use experimental calibration or some other source of calibration against lower level ground truth, if you want to call it that. That could actually also be coming back to that example. Like it could be like the convictions or insights from, from people who work closely with the market on tactical levels. It can be the output of trusted attribution models or of even attribution models provided by ad platforms. And ideally of course it's experiments and they're ideally randomized control trials. So RCTs which really allow you to get an estimate of your incremental effect of the advertising and then you want to use that to Calibrate your MMM and sort of make sure that they agree at least directionally. Right. You don't want your MMM to say like this channel has negative roas for us. And then you run an experiment, you see like hey, actually we're making 30% plus on this channel. And Igor Skokan and Harpreet Patter and I, we wrote a thing about this for Harvard Business Review In I think 2000, I believe, where we talked through experimental calibration of MMM a bit. We kind of gave the article somewhat ironic title like A new gold standard for advertising measurement, I think. And the point kind of was that deterministic attribution and therefore also to some user level experimentation in advertising is becoming harder because of privacy initiatives. And therefore the new gold standard is kind of to run experiments whenever you can and then you calibrate your MMM with that and there's different ways you can calibrate and we run through that in that article a little bit. You can just see like do the MMM and the experiment directionally agree. And if so, yes, that's okay. You can use the experiment to choose between different models you may have. Right. Like if you have different competing MMM models, you could select which one works and then in the most rigorous way you really use the output to calibrate. This can happen in a Bayesian framing through priors for the mmm. This is kind of if we talk about or like the current open source packages available for mmm. Right. Like this is kind of the way that Meridian and PYMC would go where the experimental results enter through priors and in Robin it's through multi objective optimization. So there it would be more that kind of you can add an additional objective to the optimization that then makes sure that the statistical fit so RMSE and fit against experimental results. Agree. And one thing to mention there though, and I don't want to make this too technical a conversation, but like we.
- [36:17]
Alison Schiff
Crossed that Rubicon, go for it.
- [36:18]
Julian Runge
Oh, okay. I'm sorry, should I keep going or.
- [36:20]
Alison Schiff
Do you want finish your thought? Yeah, yeah, I want to hear.
- [36:23]
Julian Runge
Okay, well so you know, in many instances what I just described would, would assume like kind of non dynamic coefficients for the advertising effectiveness or let's say static across the time period that you, you estimate your model over. That's also an assumption, right? Because in reality often your advertising effectiveness for different media will fluctuate over time. And so if you want to really think about this in a, in a more rigorous, in a very rigorous way, you ideally want to use different experimental results from different points in time and then fit, you know, your dynamic or time varying coefficients to these results if they vary over time. And this gets definitely more advanced, I think in the open source packages, the only package that currently allows for time varying coefficients, I believe is pymc actually. And the other ones don't do that because it is a bit of a, you know, it's a next step thing. It's not what you want to do when you first start using mmm. Yeah, but anyways, let's say brief, brief technical.
- [37:30]
Alison Schiff
Side note, I'm not familiar with that one with Pym3. I know Robin, I mean that's Meta's open source MMM offering and Meridian is Google's. But what is the other one?
- [37:44]
Julian Runge
Yeah, it's called pymc. I think it's actually older than both Meridian and Robin, if I'm not mistaken. Started somewhere in the 2000 and tens. Yeah. And it's an open source package for Bayesian hierarchical modeling, if I'm not mistaken. To be honest, I'm not an expert on it. So you can also use it for other things than mmm, but it lends itself pretty well for MMM as well.
- [38:10]
Alison Schiff
It seems like to do true experimentation though, you really have to be willing as a marketer to be like disabused of your notions. Right. Like to learn that something you've been doing isn't what you should still be doing and to be okay with that. And it's also, I mean, not that people are necessarily doing this, but I mean, you could. It's also possible to tweak an attribution window until a model shows that like TV ads are the top performer. If maybe TV ads are what you want to spend your money on. So if you want to live in reality, you have to open your eyes to the results. Right. That you have, you know, like a real embrace of what you're learning and you're not discarding stuff that doesn't fit into what you just want to do.
- [38:55]
Julian Runge
Yeah, I agree. One, one piece of thinking I'd offer up or in that regard is actually from entrepreneurship research. So there's this professor, her name is Sara Saraswati, I believe she's at Darden, so at the University of Virginia. And she has, she introduced this line of thinking to distinguish between effectuation and causation in terms of how people working in business, their cognitive styles, if you so will. And she found out, or like she came up with this distinction when she compared how MBA students and actual entrepreneurs were reasoning and MBA students would much more think in the causation way of things where it's more about I want to gather data, I want to create this causal model of the market and understand what is causing what and then use that to gain control. True entrepreneurs, on the other hand, use the effectuation approach. They work much more by trying to gain control over the market right now. And it's almost like a bit more of an intuitive way of approaching things. And so why I mentioned this is like, because that is the trade off that I think is also behind like when people are hesitant to buy into new data. Right. Like assume you have been doing this for 20, 30 years and you know, you've worked out this effectuation mindset. You just, you're this expert manager who knows how to run marketing and then you get causation data points coming in that challenge. What you've been doing for so long, that's hard. And I think the thing often goes wrong when people then become immune to listening to these new data points coming in. And so one way to. I think it's almost like you should maybe set flags for yourself to make sure you don't fall into that trap and challenge your own preconceptions and maybe earmark a portion of your budget to do more experimental stuff. Be in a dialogue across different hierarchical levels and also different functional levels. What I mentioned earlier, right. Like, talk to some tactical people from time to time, get a download on what is happening in the market right now and what they're seeing. And I think, you know, develop basically like a small set of heuristics that you use to challenge yourself to make sure. Right. Like, because it's true with your effectuation mindset, you've become this really good marketing executive. But then you want to be sure you stay open enough to keep succeeding. I guess.
- [41:24]
Alison Schiff
In that vein, what do you see as like the most persistent myth or misconception about advertising effectiveness that just like won't go away and still influences how campaigns are measured today? And you can't say last click because that's the obvious answer.
- [41:43]
Julian Runge
This is such a tough question in my opinion. So. Right. Like, because I think games as marketing media is one thing. Experimentation and those are definitely two. Well, more than pet peeves of mine. They're like things I really want to influence what, what's happening in the market in that regard. And I think I'll, I'll say like games and the effectiveness of games and what's possible in games. This is where I still to see like, you know, not enough happening. And maybe this is more Normative than descriptive. Like maybe I want to just. I want this very much. But yeah, it encouraged people to challenge how they think about games. I also just gave a keynote about that actually that I titled Games are marketing media, which first sounds a bit odd and many gamers actually wouldn't love hearing that. But I think it's important we start thinking about this more. Yeah. So let's make it about ad effectiveness in games.
- [42:40]
Alison Schiff
Let's. So we're nearing the end of our time and I wanted to do a brief lightning round. If you're. If you're down to do that, just one sentence or even one or two word answers only.
- [42:55]
Julian Runge
Okay, let's try that.
- [42:57]
Alison Schiff
Okay. So as previously established, you are a behavioral economist. What is a very brief example of an aspect of consumer behavior or decision making that measurement models consistently overlook?
- [43:15]
Julian Runge
Oh, wow. Well, good measurement models wouldn't overlook it. Right. That's kind of the whole remit of behavioral economics that you use psychology and the, you know, other theories to model the what first seems like irrational behavior of consumers. Because there are generalizations also possible in, in that regard. And I'm actually just teaching a class here at Medill to the professional integrated marketing communication students that I call engagement engineering. And we just spent three modules working through key concepts and one of the modules is on behavioral science and behavioral economics. And what are key concepts there that we do need to be aware of as marketers to make sure measurement and also using algorithms to shape user experiences. Work, work appropriately.
- [44:08]
Alison Schiff
Okay, well that wasn't a fair question to ask in a lightning round, I'll give you.
- [44:13]
Julian Runge
That answer was too long too, I guess.
- [44:16]
Alison Schiff
Okay, so we're going to start again. What emerging metric excites you the most for the future of advertising measurement?
- [44:28]
Julian Runge
Wow. I don't know if it's emerging, but I think dwell time is a really cool measure of, you know, ad effectiveness. How much time do people spend looking at your ad? It's certainly something that you can really only observe in digital media. Well, but everything is becoming more digitized, so yeah. Dwell time. Love that one.
- [44:47]
Alison Schiff
Interesting. I think of dwell time too as being a double edged sword because maybe someone is lingering because they don't understand the ad or they're trying to figure out where the X is or they hate it. That's also possible.
- [44:59]
Julian Runge
Yes. And that's why I find it so exciting and it's so true. There's this inverse U shaped relationship between dwell time and ad effect. Yes.
- [45:07]
Alison Schiff
Will Last click actually ever die?
- [45:11]
Julian Runge
I hope not. You know, like generally like, it's still, it's been vastly useful to so many measures and problems and attribution challenges. One thing that I think is really interesting at the moment, what's going to happen is like this whole idea of cross property, user level attribution. What's going to happen there, you know, in the end, like, are the cookies truly crumbling or not? And I think we've first seen a lot happening in that direction, right, with like Apple's att. But then currently France and Germany are challenging att and we see that Google is maybe not deprecating cookies. So I think, yeah, I'm really excited to see what's going to happen there.
- [45:57]
Alison Schiff
What is a vanity metric that you would like to just kill with fire? Like, you just want to see it retired for good?
- [46:07]
Julian Runge
Wow. I don't think I have one, to be honest. You know, I like all data and measures. There's some that need to be deprioritized or shouldn't get much attention, but I would never really think we need to throw anything overboard because you never know what might even be useful in the future as a guardrail metric or something of the sort.
- [46:29]
Alison Schiff
What would you deprioritize? That people prioritize too much.
- [46:36]
Julian Runge
Yeah. So impressions. I don't know if I want to. I don't want to deprioritize them. That's not right. But just like understanding how impressions are actually different across different media and that they're also not always much better to feed into an MMM compared to spend is something I'd at least want to discuss a little with people in the market.
- [47:03]
Alison Schiff
If you could ask every CMO in the world to run a single marketing measurement experiment, what would it be?
- [47:12]
Julian Runge
Run a branding campaign in the game and measure it experimentally.
- [47:17]
Alison Schiff
All right. Do you hear that? Cmos? Julian wants you focus on gaming, for God's sakes. Okay, so the lightning round is over and I have a last question. If you had the power to redesign the ad industry's approach to measurement from scratch, like we hadn't messed things up yet, what would it look like?
- [47:43]
Julian Runge
Wow. A good but yeah, pretty tough question in my opinion. So I think there's benefits to separating optimization and measurement. It's something I've recently talked about a little bit with Hannah Pavlov. We used to work together at Facebook and I think that's beneficial and that should always be a key pillar of how we want to think about the ad industry. And of course, if we had measurement solutions that would work somewhat agnostically across media that just be amazing? Because I think it would also help address this whole thing I've been saying about games throughout our conversation today a lot. So I don't know that I have like, you know, markets evolve and I love markets. But if, if I had like the power to, to put something in the market, it would be like a platform agnostic measurement solution that's actually reliable and trusted.
- [48:44]
Alison Schiff
Yeah, well, I mean, maybe we don't have the power to make that happen today, but you do have the power to make certain observations about the reality of the ecosystem we live in. And like you were saying, people spend inordinate amounts of time playing games. So marketers help. Help Julian sleep better at night, spend more in games, experiment with games if you want to reach roughly one third of the world's population. Is that fair?
- [49:14]
Julian Runge
Yeah, sounds good.
- [49:23]
Sarah Sluice
This episode was brought to you by Mfinex, pioneers in AI powered market mix modeling. Get fast answers to hard questions with Mutinex. You can ask for a demo@mutinex.co that's m u t I n e x co.
- [49:42]
Alison Schiff
Sat.