← The Commerce Collective Podcast
The Commerce Collective Podcast
Behind the metrics: A framework for measuring advertising effectiveness with the Amazon Ads' Analytics & Insights team
00:29:01
Loading summary
Transcript58 lines
- [00:00]
A
We have these science based approaches to determining what's the optimal ad spend and how you can use your advertising budgets effectively. As we said, it's not about spending more, but spending smarter by identifying these levers and making sure that you're spending your money in the right places.
- [00:22]
B
Welcome back to the Commerce Collective podcast brought to you by Flywheel. My name is Emma Erwin, your host for this series and I am so excited to bring you today's episode all about measuring advertising effectiveness. You might be thinking more content on measuring advertising effectiveness. To which I say, heck yes. And today's content is unique because we're getting the POV from the analytics and Insights team at Amazon Ads. Yeah, the data scientists themselves. And it's all beautifully tied together in a four part framework which we're going to cover every component of in this episode from efficiency and the impressions tipping point, the relative effectiveness of ads, ad carryover and off, Amazon impact and omnichannel measurement. Even better, the Flywheel team collaborated with Amazon Ads on metricing and process as this framework was built out. So let's meet our guests and really get into it.
- [01:08]
A
I'm so excited to be here. My name is Inigo Gutierrez Fernandez. I'm an analytics and insights associate principal at Amazon Ads. So for a bit of context, the analytics and Insights team within Amazon Ads. Our vision is to create strategic insights that enable our partner advertisers and agencies to make optimal marketing decisions. We are a group of marketing strategies, data scientists and consultants that help advertisers grow their brands both on and off Amazon through advertising. My particular background is in applied mathematics and I started my career in management consulting. After completing my MBA at Stanford University, I entered into the world of retail media and advertising technology where I spent a few years focused on helping brands make smarter data driven decisions.
- [01:55]
B
Okay, so you're just like really impressive. I feel an honor to be in your presence.
- [02:00]
A
No, no, no, no, no. It's all fancy titles and words.
- [02:03]
B
But hey, you know when I went off to college they did like a math assessment as to put you because I still had to fulfill math requirements and I made it into like remedial algebra as my entry level math for college. There's a reason that I talk into a microphone and I'm not the person doing your job. Beautiful. Okay, something we ask everyone, what is the last thing that you have purchased from Amazon?
- [02:28]
A
So I recently got a picture frame for the watercolor painting that I got and then I also have a subscription to leave in conditioner to take care of my curls. Because otherwise they get very unruly.
- [02:39]
B
I see. So you're doing the subscribe and save for the leave in conditioner.
- [02:43]
A
Yeah, I know I'm going to need it like every single month. So it just like takes that off my to do list and I know I'm always going to have it, which saves my life.
- [02:51]
B
We love the subscribe and save in my house. Lots of dog food on the subscribe and Save and like air filters for all the shedding that my dogs do and different vacuum things. So that's where we're at on my end.
- [03:05]
A
I love it.
- [03:06]
B
There are no shortages of places to invest as an advertiser up and down the funnel, on and off Amazon. In fact, it can be incredibly overwhelming. We know this and it's all unique by brand and category, as I mentioned in the intro. Thankfully, the analytics and insights team at Amazon Ads built a framework for measuring advertising effectiveness, which altogether serves as a guide for how to best allocate your budget. Let's kick this off by diving into the first component of the framework, which is efficiency and the impressions tipping point. I know that Amazon Tipping point methodology is a developing method for measuring effectiveness, but like, can you tell me more about the methodology and how it measures advertising effectiveness? Like, what is the tipping point?
- [03:45]
A
So this methodology in particular studies an advertiser's performance over a long period of time to identify trends on when their ads are most efficient. So we look at what we call a tipping point, which is what the methodology is named, after which you can think of as critical mass. So it is the level of daily impressions after which we know a campaign is going to start delivering significant results. We also try to find a plateau point, which is the opposite, which is a higher point. Or you can think of it as a point of diminishing returns, after which every additional dollar spent on that ad type is going to drive slightly lower returns than the previous dollar. This goes beyond traditional descriptive analysis. We use a statistical methodology called Regression Discontinuity Design, or RDD for short, which helps us identify the exact impression point where ad impressions start making that significant uplift based on each specific advertiser's historical performance.
- [04:43]
B
Gotcha. Can you help me understand even deeper by maybe, like, putting this into practice? Can you give me a real life kind of example?
- [04:51]
A
Absolutely. Think of it about like Goldilocks trying to find the sweet spot in the suite. You know, you don't want it to be too cold, you don't want it to be too hot. So ad campaigns can be the same. You don't want to have like, two little impressions where they're not driving significant outputs. But you also don't want to spend all your money in one ad type campaign where you get to a point where ads are not driving the results that you want. So we want to find the sweet spot to identify in which range campaigns are really taking off and start delivering significant results. It's not just about throwing more money into advertising. It is about finding this critical threshold where your investment can start delivering meaningful results. We use visualization techniques and polynomial regression to identify where this threshold has been observed for a specific advertiser and where there is statistical evidence that when an advertiser delivers impressions above a certain level, they get significantly better results consistently than when they deliver right below that level.
- [05:51]
B
Gotcha. Oh, my goodness. Regression just scarred me a little bit. I went flashbacking to, like a classroom and trying to understand regression, but that's a good example of, like, how the things that you learn can actually be applied later on in life. And that maybe that was important, that somebody taught me that. Very good. Okay, I'm curious. We've been talking about impressions, and I know impressions doesn't necessarily equate upper funnel, but is the tipping point purely looking at your upper funnel efforts on and off Amazon?
- [06:21]
A
No, not necessarily. And I think that's a great question and great clarification. So we use impressions as the independent variable for upper funnel ad types, but for other ad types, like sponsored products, we can use clicks as the main predictor metric to better align with the nature of each ad type. So, as you know, for some ad types, you're going to be paying for impression, for others you're going to be paying for click. And so we want to give advertisers a recommendation on this lever they can control through their budget, which is the number of impressions or clicks they're going to get to try to influence the outcome of sales that they're trying to get. So this can be personalized for each ad type based on the nature of each ad type.
- [06:59]
B
Got it. Understood. I think that's a pretty good overview of the tipping point methodology. And so I'm going to move US into component 2 of the framework for measuring advertising effectiveness, which is, well, effectiveness and the relative effectiveness of your ads. And I think, you know, we talk all the time about effectiveness because it's such a prominent concept. Like, every marketer wants their efforts to be effective, but a lot of advertisers are left saying, like, did these efforts actually result in a sale or would that sale have happened without any spend and it can be really, really tough to prove. So how's your team thinking about all of this and the relative effectiveness of ads?
- [07:35]
A
I fully agree with that. I know for advertisers it's very important to understand the effectiveness of their ads. So what we do at Amazon is we look at all variables that affect the effectiveness of sales and how they each influence each other to determine how how each of the variables is driving relative effectiveness compared to the others. So think of it as a map, let's say where like all the different variables that might be influencing revenue are mapped out and we have connections between each of the variables and revenue. And what we do with this analysis is we measure the effect of each of these different variables on driving total revenue and how strong that effect is. And to do this we use a science based model called causal directed acyclic graphs or DAGs for short. This is a methodology that was co published with Data Sciences team and researchers within Amazon Web Services. It helps us model these cause and effect relationships between the variables that can be ad spending for all the different ad types, detailed page views, glance views, search terms, search trends and how each of them are driving impact on the total sales.
- [08:51]
B
Goodness gracious. Oh my goodness. Okay, I'm going to again ask you to help help me simplify this. You know like how do these. Did you say DAG or DAG models?
- [09:02]
A
Yeah, DAG or DAG both work. Exactly.
- [09:05]
B
How do the DAG models actually like tie to sales in a way that your standard marketer such as myself can understand?
- [09:12]
A
Of course, let's narrow it down to two ad types. For example, imagine you're trying to understand the effect of two different ad products, let's say search ads and display ads, and how they contribute to your overall retail revenue on Amazon. Using these dags, we map out each of the ad types and what is their effect on sales and calculate the percentage of contribution of each of the ad types on driving changes in revenue. And we do this through sorting data over a very long period of time and while controlling for other brands past performance other external variables to make sure that the impact we're estimating is fully driven by the ad investment and not by any other external factors. So we isolate that part and by studying both of them together, we can also study how they each influence each other. And at the end we get a number on what percentage of the incremental sales was driven by each of the ad types during the period of time we studied.
- [10:10]
B
Gotcha. And I'm going To even when I was reading through your answers to these original questions, I kind of was thinking of myself as like an average shopper on an average day, removing myself from really knowing all that much about advertising. But, like, when it comes to purchasing a product that I don't purchase within 1 second of looking at it for the first time, like, I know that there are a million subconscious thoughts coming going on in my head. You know, I see ads here and there, but am I really registering that I have seen those ads? And so my question really is like, what are the different variables that this model looks at? And how does the model determine if me interacting with an ad at some point was intentional or just a complete coincidence that I eventually made a purchase?
- [10:46]
A
Absolutely. And that's a great question and I fully agree with you. Like people's thought process and decision making, it's a very complex topic. But in this methodology, we are not trying to predict or understand individual behaviors, rather measure broad outcomes. And that is a cool part of this methodology. It doesn't try to explain the actual process or the thought process behind it, just measure what actually happens, whether the purchase happened or not. So we look at a brand's ad investment and sales data over a long period of time as the main inputs and ad investment, like, broken down by each of the different ad types. And if we constantly see over time that when there's a change in the investment on one of the ad types, it is followed by a change in sales, even when accounting for things like seasonality, tentpole events and promotions and other things, then we can come to a conclusion that changing the level of investment of that ad type is going to drive more sales. And when we compare that effect for all the different ad types we're studying, we can compare the relative effectiveness of each of the ad types at driving sales.
- [11:53]
B
Amazing. That makes sense. You're not trying to be inside my head reading my thoughts, and that's probably for the better. So it's probably more effective. And for on the topic of effectiveness, more effective to not be trying to understand why I'm doing every single little. Having little every little thought that I don't even know that I'm having. Exactly.
- [12:11]
A
We'll leave that to psychology.
- [12:13]
B
Yes. I was gonna say I was gonna crack the joke. It's like there's a whole field that tries to understand that.
- [12:19]
A
Exactly. Exactly.
- [12:21]
B
Amazing. I'm going to move us into component three. We just covered relative effectiveness of your ads, and before that, the tipping point methodology. And now we're talking about number three. Which is ad carryover or the long term effective ads. Can you tell me about like what does that actually mean? What is ad carryover and why is that important in the framework?
- [12:41]
A
Of course, and I know carryover might be a hard to understand word, but as you said, it's the long term effectiveness of ads. So with this metric we are hoping to provide advertisers with a new perspective on how upper funnel versus lower funnel ad products work in the long term by measuring their impact beyond the standard 14 day attribution window. So some effects might be immediate as you said from users like you see an ad, you purchase the product immediately while watching the ad, while others might purchase the ad weeks after seeing others might purchase the product weeks after seeing the ad. Our model can tell us which ad products have a longer term effect and how big that effect is. We normally examine long term impact in two week intervals up to 12 weeks after the ad exposure. This is crucial because advertising isn't always about immediate results. As you know, most marketers have access to a 14 day attribution from standardized reporting tools and use that to compare ad types and their performance. But that doesn't always tell the full story. So to measure carryover effect we use sophisticated time series modeling approach called Varmax, which has significant advantages over more traditional methods in that it recognizes that an advertiser seeing certain results might affect how they're investing in the future. And so it accounts for this correlation of the outcomes with the inputs and we account for that. This approach considers the expanded customer journey metrics beyond detailed page views and conversions and includes things like branded searches and brand loyalty to understand how our ads having an impact beyond the regular 14 days of attribution.
- [14:24]
B
Gotcha. When I hear the word long term effectiveness of your advertising efforts, I immediately go to amc. So I'm curious like how this model that you're working off of differs from what brands who have some SQL savvy people can like uncover via analyzing data. Amc. Is there a use case for one or the other? Help me understand that.
- [14:45]
A
Of course, and that is a great point as they're both trying to tell you what's the long term effect of ads. I would say the main difference with the AMC long term metric is that in AMC we're just doing a descriptive analysis and it's not controlling for external confounders like additional ad exposure. So to simplify it a little bit with an example, let's say I was exposed to a sponsored product ad today and then I make a purchase of that product in December. So in AMC's long term sales metrics, that sale in December is going to be attributed to my exposure today. If I'm studying the long term effect of today's ads, even if I had additional ad exposures in November to STV and display campaigns and another sponsored product ad in December before purchasing the product. Right. So the December sale is attributed to whatever time period you're starting with a long term sales metric. With this ad carryover analysis, what we do is we remove all of this external noise to isolate the impact of each specific ad type and identify how the sales impact spread across the two to 12 weeks after the sales impression. So in the ad carryover effect we would know that I was exposed to the ad in January, but we would also account that I was exposed to a different ad in November and a different ad in December. And so those are probably going to have more weight in me making the purchase because it's closer to the moment when I, when I purchase the item. And so if we see that constantly purchases are happening like 12 weeks after someone was exposed to a certain ad type, we can say that statistically there is a lot of evidence that that ad has a longer term impact.
- [16:22]
B
With everything that you talked about, could you also give me an example of like how long term effectiveness is reported back to a brand? Like what is the metric for long term effectiveness? Is it monetary value attributed to an ad type? What is the, what are you giving back to brands so that they understand what you're saying?
- [16:38]
A
So we normally report it back as a 100% broken bar bar, different time frames. So for example, you have like zero to two weeks, two to four weeks, six to eight weeks, etc. And we measure what percentage of the total effect of that ad happen within each week time frame. So for example, if you have a NAT type that has a very short time effect, you're going to see 100% of the effect happening within the 0 to 2 week range. Very similar to the 14 day attribution window. But with a NAT type that has a much longer effect, you might see that only 20% of the total effect happened within the first two weeks and then another 40% of the effect happened two to four weeks afterwards. And then the last 40% happen in a six to eight week time frame. So this means that when you're comparing the 14 day ROAS of both AD types, you should consider that the long term effect of the second AD type is only really reporting 20% of the total ROAS effect of that ad type. To make for a more fair comparison of the impact of each ad type.
- [17:39]
B
Gotcha. And so now that we've said ad type a bunch of times, I'm going to ask you, can you just like level set with everyone? What's an ad type that should in theory have long term effectiveness versus one that maybe has short term effectiveness? You know, just like, just give the people the baseline information.
- [17:55]
A
Yeah, I'll give them a scoop behind the scenes. Of course, like this can change a lot by brand and by category, you know, but it is much more common for us to see that sponsored product ads have a more immediate effect and most of their effect is realized within the first two weeks. Whether a streaming TV ad can have a longer effect and keeps showing purchases and having effect like many weeks after the ad was seen.
- [18:21]
B
Gotcha. That makes sense. Thank you for describing. Okay, I have one breakup. Fun question for you. When you graduated, did you ever think you'd be on it, like doing a podcast describing how you're measuring the effectiveness of ads? Like, was that ever in your mind of what you wanted to do?
- [18:36]
A
I honestly never thought it was going to be specifically for advertising. I feel like I discovered that recently. But I been so enamored with the industry because it is very data driven. And so like when I was studying math, I knew I loved data, I knew I loved discovering patterns and figuring out how all variables related to each other. But I was never really exposed to advertising as an industry. And after joining I realized it has everything that I like about complex systems and different things relating to each other and trying to clear the noise to identify those trends and then also being very, I would say, like realistic and applicable to the world. Because I knew I never wanted to go into academia. That was like way too abstract for me and way too detached from reality. But here's like we identify the trends and then we can act upon them and so we can give a recommendation and budget, we can give a recommendation on campaign and we can see the results and measure the results afterwards and keep adjusting. So that has been really fun. I would say unexpected to be doing this, but it feels like the perfect place for me to be.
- [19:39]
B
Amazing. I love it. I'm moving us to the last component of the framework. We're almost there. We've almost done it. And that is, and I think this one's really interesting that it is off Amazon Impact and Omnichannel Measurement. And so maybe the best place to start here is with something I've heard of called like the Amazon Shopper Panel. Can you help me, I'm assuming these things tie together. Can you help me kind of understand what we mean by off Amazon impact and omnichannel measurement via the Amazon Shopper Panel?
- [20:06]
A
Of course. And I think this is really exciting tool that Amazon has been working on for a while. So the Amazon Shopper Panel is an invitation only program where participants earn rewards for sharing their shopping receipts from outside of Amazon. We have almost half a million active monthly participants and they share more than 3 million receipts monthly. And so this gives us unprecedented insight into omnichannel shopping behavior. Using this, advertisers can understand the true total return on ad spend by measuring sales impact across all channels. This is crucial because, as you know, customer journeys are not linear anymore. Some might see an ad on Amazon but make a purchase in a brick and mortar store and that doesn't mean the ad was not, not effective for the brand.
- [20:55]
B
Gotcha. I was going to be cheeky and just say, where's my invitation to the Amazon Shopper Panel? I feel like I'd be pretty interesting, but whatever. Okay. More importantly, how do advertisers access these insights? This seems pretty cool. Is that also invitation only?
- [21:14]
A
No, I agree. This is a very, very cool feature. These insights can be incredibly valuable and help brands inform their advertising strategy. So brands can talk to their account teams at Amazon and they'll work with my team, the analytics and insights team, and we can teach them more about these insights and how to get started on using them.
- [21:33]
B
Perfect. Okay, and so now I'm going to try and talk about data here and I want to, like, I'm curious, how are you actually connecting those advertising efforts to transactions that occur via other retailers? And I'm going to like present you a scenario. You tell me if I'm thinking about things correctly, like, let's say an electric toothbrush brand knows like my customer id, or better known as Me. And I know that's like a piece of information that Amazon provides advertisers so that they can actually see what people are doing. But let's say I've been searching for electric toothbrushes on Amazon and I'm naturally getting advertised to on Amazon.com, but then I'll upload a receipt that says I bought a toothbrush at Target. Is it because of that, like, connective point of my customer ID that you're able to say, hey, those Amazon ads led to a purchase at Target most likely. And like, that's still a win for your brand overall. But how are you actually connecting those data points?
- [22:24]
A
Yeah, That's a great question. So we do basically an AB test. We look at two different audiences and so we have an audience of everyone that was exposed to an ad, everyone that wasn't exposed to the same ad. And we can then measure the statistical difference in purchase rates of a brand in these two groups based on their self reported purchasing and tickets, and we can estimate how big that difference is. So we look at everything like at a group level and we just compare the trends amongst groups to see if there's any difference in their behaviors.
- [22:58]
B
Gotcha. I see. Okay, we've covered a lot here. We've covered a lot of data science. Oh my goodness. With everything we've talked about, how does an advertiser actually put all of this together and budget more effectively? Is there anywhere in this framework that's more important than the rest? How do you actually take all of this information and say, okay, now what?
- [23:18]
A
I know it can be a little bit daunting to see all these things coming in at once and be like, how do I action on them? I think for me the key is to use them holistically and you can also do it progressively. So for me, the starting point would be the tipping point analysis that can help you determine if your current campaigns are running at that minimum effectiveness spending levels. You can adjust that in a very short term to get your campaigns to that level where they're driving the maximum effectiveness that they could be driving. Then once you've accounted for that, you can use the other parts of the analysis, the relative effectiveness and the carryover effect, to understand how each app type is providing benefits for your brand and how they each relate to each other. And that might help you create a budget that aligns with your outcomes, that considers not just immediate roas, but considers the long term effect of each of the ad types and the relative effectiveness of driving sales. And then lastly, the Omnichannel study can help you understand how your advertising efforts within Amazon are driving sales elsewhere, just to make sure that you have the full impact of your campaigns and you properly measure how much sales you were driving for your brand.
- [24:37]
B
Got it. Is there one component that to you as a data scientist, is there one component that is the most interesting to you in terms of what the team has built and what you're looking at?
- [24:47]
A
Absolutely. I mean, they're all really fun to work at, but I think from my experience and from working with brands, I think the tipping point analysis is super powerful because it is very immediately actionable and it gives a very clear recommendation. It literally Shows you for this ad type, based on your specific performance, you should be in between here and here to achieve maximum results. And that's something that can be very powerful for brands to make sure that they're maximizing their outcomes and they can test it immediately and then keep adjusting based on that. And it's also not too restrictive. Like you get a range of impressions where you should be playing, but it's not giving you like you just need to do exactly this and don't move from here. So it still leaves a little bit of room for experimentation to like test it out and get to that like optimal thing that aligns with your strategy and the outcomes you want to get to.
- [25:38]
B
I love that. That's cool. I was like, I bet it's going to be the tipping point. That one, that one's pretty cool. And I'm sure a lot of brands get that information back and they're like, oh wow, that's actually like life changing. And thank you, thank you for pointing out that we maybe were not hitting the correct point point or we've gone too far, we went over the tipping point and we're spending all these dollars that aren't actually doing anything.
- [25:59]
A
Yes, it has been very eye opening for a lot of brands.
- [26:03]
B
Okay, if you had to consolidate key takeaways from this discussion, what would you leave the audience with?
- [26:09]
A
So I highlight three main points. First, we have the science based approaches to determining what's the optimal ad spend and how you can use your advertising budgets effectively. As we said, it's not about spending more, but spending smarter by identifying these levers and making sure that you're spending your money in the right places. Second, I would say when thinking about ad types, always consider both short term and long term effective ads. What might look like underperformance for a certain ad type might be building valuable long term brand equity that can be very important for a brand to grow in the future. So always think about that and not just stick to the short term 14 day results because that might be a little bit too short sighted. And then third, don't look at channels in isolation. We live in a very complex world where things are not linear and people are jumping from one place to the other. So the Amazon shopper panel can be very valuable. But to understand how your advertising is having impact beyond Amazon, just to inform your strategy and making sure that you're getting the most out of it.
- [27:19]
B
Got it. Boom. Okay, I have one very last question for you and this is another just fun one that we ask everyone because the answers are always so interesting. So I'm going to have you tell me what is something that lives on your digital wish list that means it's just in a cart in a tab somewhere that you just won't purchase and why?
- [27:39]
A
I hope I don't fall in the boring category, but I've had a robot vacuum on my Amazon wishlist for a few months. But honestly, I always stop myself from purchasing it because it takes like five minutes to vacuum my tiny apartment. So I think it would just be like, way too lazy to have like a little robot vacuum running around to avoid, like, vacuuming in like five minutes every couple of days. But sometimes I feel like I should just do it and I might get it, so I don't know, I might end up kicking.
- [28:07]
B
Vacuuming sucks. Like, as someone who has to vacuum every day, it just. It sucks. And unfortunately, I don't think I have a husky. So I don't think a robot vacuum is going to do, like, I need, like an industrial. But that's a good answer. I like, it's more of a practical thing. It's not necessarily the cost. It's like, no, like, I could. I could vacuum my own apartment, you know, but also a robot could.
- [28:32]
A
Yes. Things get dusty, like, so quickly here. So, yeah, who knows? Maybe I'll do it.
- [28:41]
B
And that wraps up another episode of the Commerce Collective podcast. Massive shout out to Inigo and the Amazon Ads team for all of the work here and investment into making sure that advertisers really can see the impact of their investments, all backed by Data Science. I've been your host, Emma Irwin, and we'll see you next time.