← The Analytics Power Hour
The Analytics Power Hour
#277: ANOVA? I Hardly Know Ya'! with Chelsea Parlett-Pelleriti
01:00:29
Did you know that, upon closer inspection, many a statistical test will reveal that ()? That wound up being a key point that our go-to statistician, , made early and often on this episode, which is the next installment in our informally recurring...
Loading summary
Transcript131 lines
- [00:00]
Unknown Host
Foreign. Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language.
- [00:14]
Michael Helbling
Hey, everyone, welcome. It's the Analytics Power Hour. This is episode 277. Given the chance, we'll frequently take any excuse to do just whatever we feel like doing. But ostensibly this episode is part of an unofficial series from our listener survey. It's another deep dive into a statistical concept. That's right. We've got two keys in anova and we're going to drive straight into a post hoc analysis of all of our life choices that led to this moment. All right, we're doing analysis of variance or anova. I really don't know much about it. It was something that the SAS developers in the other department would talk about sometimes. But I've. This one's been out of my league for a long time, so I'm pretty excited to learn a little more. Julie Hoyer, are you stoked about another stats focused episode?
- [01:08]
Julie Hoyer
Of course I am. I can't wait to get into it.
- [01:11]
Michael Helbling
Awesome. And Tim Wilson, I know you're probably raring to go.
- [01:16]
Tim Wilson
I'm looking forward to being more confused coming out of this than I am going in.
- [01:21]
Michael Helbling
Well, I don't know if that's the result we're looking for, but we'll see if there's any significant difference in your knowledge after the fact. All right, I'm Michael Helbling, and for our guest, well, we had to do it. We had to bring back our favorite chatistician from just 10 episodes ago, and it's her third time on the show. Chelsea Parlette Pelaridi. She's a consulting statistician at Recast, and She uses her PhD in computational data science. She teaches statistics and math at Chapman University. And once again, she is our guest. Welcome back again, Chelsea.
- [01:53]
Chelsea Parlette Pelaridi
Thank you so much. I'll need to do something notable before the next time I come on. So you have a good bio for me that's different than this one?
- [02:01]
Michael Helbling
Well, you know, we'll do a. We need to go deeper into, like, what your interests are. So, like, you know, corgis, Stardew Valley, those kinds of things. And we can start to put. Put a bio together around that stuff.
- [02:13]
Chelsea Parlette Pelaridi
That'll come up today, actually.
- [02:15]
Tim Wilson
Perfect.
- [02:16]
Michael Helbling
So it's not coming from me, but I think a good way to start this process is maybe start at the very beginning of analysis of variants and even maybe start with how you feel about it, because I know it's not your favorite thing.
- [02:34]
Tim Wilson
It wasn't a topic that Chelsea pitched us.
- [02:37]
Michael Helbling
Yeah, yeah, we were like, hey, please, can you come talk about this?
- [02:42]
Chelsea Parlette Pelaridi
Yes. Well, I'd actually like to start with a poem, if you don't mind.
- [02:47]
Tim Wilson
I love it.
- [02:48]
Michael Helbling
Yeah, that's right up my. Exactly the kind of start I think we need here.
- [02:53]
Chelsea Parlette Pelaridi
Perfect. Okay. So this kind of encapsulates how I feel. And I want to clarify, I don't have a problem with Villanova itself. It is more the way we communicate about it that sort of distracts people from things that are good about Villanova. So I'm not wholly against, but if I may, from the archives of my Twitter account, May of 2020, there once was a model ANOVA who, along with their cousin Ancova, made a great big confession. We're the same as regression, but we've established a Persona.
- [03:31]
Tim Wilson
Oh, that's a limerick.
- [03:34]
Michael Helbling
Yeah. ChatGPT could never. Yeah, could never.
- [03:38]
Chelsea Parlette Pelaridi
They could never, never come up with that. So that's the basis of how I feel about anovas, which is that they're linear models. And when we teach them as separate concepts, people sort of lose that connection. And so that's my biggest gripe with anova. It's not the actual math or anything behind it. It's that often, especially in my, you know, original field of psychology, people teach these ANOVA models, ANCOVAs, MANOVAs, you know, all the different letters that you can cram in there. And they teach them as something that's distinct from a regression model. And when you do that, people really lose two things. One is a connection to any of the really great linear regression knowledge and content that they have, and two is the generalizability of the concepts that you learn in an anova. Like, one of the things that I would run into a lot, especially in my psychology days, is people thought an ANOVA is one thing, an ANCOVA is another thing. A MANOVA is a third thing. A repeated measures ANOVA is a fourth thing. And they didn't see how they were related because they weren't taught in the linear model context. So they didn't see, like, oh, an ANCOVA is just like, you basically add a covariate to your regression model. And so that's my main issue, is that when people talk about using ANOVAs, they're typically talking about it in this framework of, like, this is a separate thing from a regression model. When really what you're doing when you fit an anova, when you use an ANOVA is you're fitting a linear model, and then you're looking at the outputs of it slightly differently. Than you might if you ran a traditional LM in R where you're running a linear model.
- [05:33]
Tim Wilson
Putting aside all of the linkages there, defining where an ANOVA is or could or should be, what's it doing? What's the purpose of that class of methodologies?
- [05:47]
Chelsea Parlette Pelaridi
Yeah, it's in the name. So an analysis of variance or an ANOVA is.
- [05:51]
Tim Wilson
Well, that just turned off the business users. You're like, come on, what more do you need to know?
- [05:56]
Chelsea Parlette Pelaridi
Yeah, of course, obviously. But it's analyzing the variance in the data. So if you think about, let's say, a data set that we could have, say you're trying to see, there's three different ad campaigns that you trialed and you're trying to figure out are they different? Are they all giving you the same click rate or are they not? Are they all giving you the same average order or are they not? And when you look at an anova, essentially what it does is it says, well, look, take the average value of the order. So like, if you have all of the order values during your experiment, you have these three different marketing campaigns say, that you sent out. One of the things that you might want to do is say, okay, there's a lot of variance here, right? Some of my orders are for $70, some of them are for $20, some of them are for $400. What can explain that difference in the orders that we see? So we're observing that orders are not all the same. Why are they different? And we basically take all of the variation that we see, right? Not everyone has the same order value. You can picture the mean order value and everyone's order value is sort of like hovering around that mean value. Some are really high, some are really low. And so what we're doing is we're partitioning that variance into sources that we care about. And the simplest case, like a one way ANOVA you often hear people talk about, the simplest case is that we have two categories of variation that we really care about. One is variation due to the group, or in this case, the marketing campaign, and then variation due to what we would call randomness. It's variation within a group. At its simplest level, the ANOVA is basically taking that variance and it's partitioning it into those groups. So what variants can we attribute to marketing campaign, what variants can we not contribute? Marketing campaign and then it compares those things. And essentially what you're doing when you're running, what we typically think of as an ANOVA is you're seeing if there's statistical significance or you could use a Bayesian framework, but usually you're using a frequentist framework. You're seeing is this statistically significant, is the amount of variance that this explains something that is notable or unexpected under the null. That's what we're doing. We're just like partitioning the variants. And variants like the ANCOVA are just adding another category. So if we have a covariate like say H order.
- [08:30]
Tim Wilson
Wait, hold on. Can we stop before we go one level?
- [08:34]
Chelsea Parlette Pelaridi
Sure, sure.
- [08:35]
Tim Wilson
So just to say, took the dumb, dumb analyst or the marketer who's just looking and says, I'm just looking at average order value and I break it down by campaign and one average order value is $75, one is 80 and one is 90. The just by looking at the, which is a mean, the average order value that there's a tendency to say, well, these are different. And it's easy to say, well that's, that's the difference between these. But everything you just described was saying, well, if order values are like all over the place and it just happened to be that you dropped in and partition them, slice them by campaign. Yeah, you just happen to get bigger ones in one and not in the other. So it's giving you a way to say given these, these observed different means, how am I confident that that means the way that I partition them actually is, is contributing to that? It's not just that. I'm just kind of arbitrarily seeing that it's, it's a noisy wide spread. Is that, am I playing that back accurately?
- [09:49]
Chelsea Parlette Pelaridi
Exactly. So like you can imagine a scenario where let's say you have this magical campaign where like everyone who gets variant A, their order is right around $80. Sometimes it's 81, sometimes it's $79, but it's right around $80. And variant B, it's right around $60. Right. Sometimes it's 61, sometimes it's 59, but always around the same. In that case, it would be super clear, even without a statistical test that you could visually plot that data out and you would see that the amount that orders vary within your campaign variance is so small compared to the amount that the two differ from each other. I think I said $20 difference between them. And that's what you're quantifying mathematically for basically cases where you can't immediately see on a graph like in the example I just described. So technically in an an, the null hypothesis that you're testing is that whatever groups you have, so it's usually Two or more. Because if you had only two, you could use a T test. But basically you're saying all of the means of these groups, however many there are, are equal. That's the null hypothesis. And the alternative hypothesis is that at least two of these means are different. And so this gets into something that maybe is too deep. You can stop me again and we'll go back. But an ANOVA is like the F test in an ANOVA that would say, okay, here I have three campaigns or I have 10 campaigns. Is there a statistically significant amount of variance explained by campaign? You're essentially doing something called an omnibus test where you're testing, is there a difference somewhere in this mess? But it won't tell you by itself, the F test will not tell you by itself where is that difference. And so the omnibus test is sort of looking overall at the variance explained when you know what campaign someone has been exposed to. Whereas typically we'll often have questions that are a little more targeted than that. Right. We want to know, okay, this is our business as usual campaign. Here's like an experimental campaign and here's like amped up version of that experimental campaign. In that case, what we really probably care about is, is business as usual different from the other two? And is our amped up experimental campaign better than the regular experimental one? And so an ANOVA by itself won't tell you that. You'd have to follow it up with post hoc tests which you mentioned in the intro. And yeah, so that's essentially what you're doing at kind of the simplest level of an anova.
- [12:35]
Julie Hoyer
So in that example, actually would you be able to just run an ANOVA on the data for the business as usual campaign and the experimental one and just do like that pairing? So you could choose the pairs of those to look at so then you could have the clear answers you're talking about. But you traditionally an anova, somebody might be like, no, we're gonna throw all three in. And to your point, the result that would come out of that ANOVA would just say if there is actual variance between those three categories, it won't tell you between which two. But with three, it's easy to be like, I'll just split it out. But to your point, like a lot of times if we have a ton of categories, it becomes very, you know, cumbersome and not realistic.
- [13:24]
Michael Helbling
Alright, let's talk data. We all love insights, but let's face it, setting up integrations, that's not exactly a party. That is why there's fivetran the smartest, easiest way to automate your data pipelines. Think of it like the ultimate set it and forget it gadget for the discerning data professional Connect. Relax and let fivetran handle the heavy lifting. Your data lands safely and swiftly into your warehouse, ready for action and analysis. Curious? I think you are. Head over to fivetran.comaph right now to stay updated on the latest news and events and see how FiveTran can make your data dreams come true. That's F I V E t r a n.comAPH Trust me, your analytics will thank you totally.
- [14:17]
Chelsea Parlette Pelaridi
Well, not only Cumber, I have so many things to say to that because that was such a good point.
- [14:22]
Tim Wilson
Well, so I think you want to hit the what if there were just two? And then you want to hit the what if there are a whole bunch?
- [14:27]
Chelsea Parlette Pelaridi
Well, if there's just two, it kind of doesn't matter what you do because you're essentially by running an anova, running a T test between them. Something I loved. I don't know why this fact was so fun to me back in the day, but when I first learned this, I learned that the F statistic you get under very specific conditions, including there's only two groups, is just the T statistic squ that you would have gotten if you had run the same type of T test instead of an anova. So there's a really one to one relationship there. But you said two things that I thought were really important. One is that it's cumbersome to run a bunch of these different comparisons, which is true, but in a sense unavoidable if you're interested in all of pairwise comparisons. But I think the point that you're kind of implying but not saying out loud is there's also a problem if you're using the frequentist framework in multiple testing. Right? Let's say I have 10 groups and I want to compare every pair of two. I can't do that in my head, but it's what, like 10? Choose 2? I don't know what that number is. Lots of comparisons that are happening and usually in a frequentist framework we're choosing at like an alpha level. So like usually we use 0.05 so 5% as our expected error rate under the null. Right? So it's like type one error rate. If there is no effect, this is how often we'll be kind of misled by the conclusions we make of the test. But if you're running 30 of those comparisons, suddenly your family wise Error rate, which is the error rate of making a mistake in that family of comparisons is huge. And so that's a problem. Another thing that is important is, yeah, we could just go, okay, filter the data, only include baseline, business as usual, and the experimental. But one the things that can be really helpful with an ANOVA is that you're actually increasing your power, statistical power that is not, you know, I don't know, what other kind of power are you increasing? You're actually increasing your power because the estimate of your error is going to be more precise with more groups. Because one of the assumptions of an ANOVA is that you have, I think it's, I think this is just heteroskedasticity, but basically you're assuming that the variance is the same across your different groups. And one of the things that that gives you, if that's true, is that you get a better estimate of what that error is if you look at all 10 groups that you have than if you truncated your data and only looked at the two groups that you, for instance, in this case are interested in. And so you're actually increasing your error power of estimation a little bit if assumptions hold, et cetera, et cetera. And so there's actually a benefit to running the ANOVA itself power wise. But also, like you're pointing out, you could partition everything, but you'd have to be more thoughtful about what comparisons you want to run and control your family wise error rate. Now, I'll take it one step back to my critique of the anova. I actually think it's better to be thoughtful about the contract. So in an anova, we usually call them contrasts, right? Like which comparisons do you want to run? And I actually think it's better to be thoughtful about that. Correct for any family wise error rate inflation that you're causing and just look at those rather than rely on the omnibus test. Unless you're actually just trying to answer the question the omnibus test answers, which is, is there variation somewhere? Are all the means not? Are not all the means equal? It's a bit of a weird way to say that. I actually think it's better to be thoughtful. And one of the things that is good about how people teach the ANOVA is usually you teach, okay, you run the omnibus test, but then you follow it up with post hoc comparisons, different pairwise comparisons you might want to know about. One of the things I really love when I was learning this back in the day is that there are some really thoughtful frameworks. You can define contrast however you want if you've ever worked with ANOVAs and R, you know, you can define your own contrast matrix. So whatever contrast you want to run, you just put them in there and it'll run it. But there's some established ones that I think are really thoughtful, and some of them have to do with sort of the example we talked about of like, okay, here's a business as usual. So in a sense, like a control group. And then here's a moderate experimental and an extreme experimental condition. There's different types of contrasts where they kind of predefine for you, what you're interested in. So I'm interested in control versus the average of the experimental. That sort of answers like, is my experimental condition working? And then I might be interested secondarily in the contrast between moderate experimental and extreme experimental, because then that tells me, like, hey, when I really take this campaign to the nth degree, force someone to click on my ad, essentially, is that actually helping compared to my more moderate, hey, click on my banner? And those contrasts are very thoughtful. It's very specific to the situation you're in. And I think my overall critique of statistics as a whole is that sometimes we encourage people to not be thoughtful. And I'm always in favor of something that encourages someone to be thoughtful. And it's not the ANOVA's fault, per se, but it can encourage people to sort of just look at the omnibus F statistic, F test, when that's not really what their question is. And because they haven't thought about it, it's just sort of this like, I Learned an Anova 5 years ago. I'm gonna throw an ANOVA at it. You really lose a lot, both of statistical power as well as kind of clear answers to your questions.
- [20:40]
Julie Hoyer
And to be a little specific, when you were saying you get more power by having more categories put into your anova, is that because to calculate the F statistic, it's comparing the variance within the groups to the variance between the groups. And so if you have more groups, you get more inputs for both of those measures. So inherently, you are getting. Getting an F statistic that's more representative of like or something. You could generalize more across the, like, the categories. Am I getting close? But I'm thinking of, like, sample size. So sample size of, like, these variance measures, you're getting more of them with the more categories that you give to the anova. So that's kind of where my brain was going, but I don't know if that's actually how that works.
- [21:31]
Chelsea Parlette Pelaridi
It's even simpler Than that, like, I think it's. You're correct. But also, even if I am just interested in category A versus category B, if I'm assuming that all of my groups have the same variance, that's something I need to estimate with my model. I don't know what the population variance is there. And if I have seven groups and like you're saying bigger sample size to estimate what that variance is, even if it doesn't help me with the between group thing, it helps me with the within group estimation, which is exactly what can happen here. But I will say that kind of relies on the assumption that they're all the same and that the pooled variance is a good estimate. And is that it was true? I don't know.
- [22:17]
Julie Hoyer
It depends.
- [22:18]
Chelsea Parlette Pelaridi
But in theory, I have two questions.
- [22:21]
Tim Wilson
And you can choose to ignore the first one if it's like that is a whole other episode, but just some fundamental intuition around what a T test is and does. And maybe it is a companion because as you're talking about, you've got a control group and an experimental group, and that doesn't necessarily have to be run in a controlled experiment. You just got different groups. But when you run a controlled experiment where you do have multiple groups in an experimental fashion, you wouldn't really use an ANOVA group. Or would you?
- [22:54]
Chelsea Parlette Pelaridi
In the example you gave, it sounds like there's only two, like an experimental and a control group. And in that case, the T test should give you roughly equivalent, if not exactly equivalent, results to an ANOVA on the same value. So I've actually run this before when I was teaching or back when I was working in psychology, where you can set up data like that and then run it through the ANOVA function in R, run it through a T test in R and you'll basically see what I said earlier is that you're going to get the same P value with rounding and computational error, and then you're going to get the T statistic, or the F statistic is T squared. So you're really answering the same question there. So in that case it does not matter.
- [23:41]
Tim Wilson
What's the approach of the T test? I get that you wind up in the same spot, but presumably if you're teaching a T test, you talk about it in a completely different way.
- [23:51]
Chelsea Parlette Pelaridi
I will say it's a different framework. And the one thing I do love about how we teach ANOVAs is that in a T test, what you're testing is the difference, the delta, between the two means. And you're comparing that to A distribution under the null and blah, blah, blah. In an anova, you're really thinking of things not as like, okay, here's a difference in means that I'm testing, but here's the variance that's explained by knowing what category someone's in compared to variance that's not explained by that. And then again, you stopped me before, but if you have an ancova, I'm going to squeeze it in. Now you can partition into a third category, which is variance due to a covariate like age or location or something like that. And so ANOVA is really focused on this partition of variance, Right? How the data points vary about the mean. Can we explain part of that variance with your category and part with randomness? Whereas a T test is mathematically like you're pointing out, exactly the same. You're going to get, under certain circumstances, basically the exact same output, but you're kind of thinking about it in a different way. Like, a T test is looking at what is the difference in these group means. Say one group mean is 10 and the other one's 5. That difference would be 5. How likely are we to get a difference of 5 if there's truly no population difference between these groups? Whereas an ANOVA is answering what is essentially the same question, but from a slightly different perspective, which is okay, if I know what group you're in, how much of the variance of the scores I'm getting? Can I explain with that? And the benefit here is that an ANOVA technically generalizes to more groups, whereas a T test, you would run pairwise T tests between them.
- [25:40]
Tim Wilson
So for the ancova, can you introduce.
- [25:43]
Chelsea Parlette Pelaridi
Multiple covariates, or you can do whatever you want. It's just a linear model, a regression.
- [25:50]
Tim Wilson
So that starts.
- [25:50]
Chelsea Parlette Pelaridi
Okay, exactly right. Okay. And so to get into that complaint, I think it's Daniela Witten, who had that series on Twitter months or maybe years ago, where she would just retweet things and say, it is just a linear model, an anova. Well, okay, let's be clear. Technically, what you're doing when you fit an ANOVA is you're fitting a linear model, and then you're using this framework of variance, the anova, the analysis of variance, to analyze the results. But at its core, what you're analyzing around is just a linear model, and you can add more covariates, you can add tons of different things. And that's my sort of problem with the framing of how we teach anov. It doesn't make it clear that that's the case. Whereas when we teach linear regression, we're a little bit better about like, yeah, throw in whatever covariates you want, throw in random effects, you know, do gam. So do like some smoothing and transform with polynomials, your predictors and then put them in and like that Flexibility is not inherent to the way that people have been communicating about ANOVAs.
- [27:00]
Julie Hoyer
And I'm having this maybe light bulb moment unless I'm really not following it off the rails here. But Tim, remember when we talked about blocking in test RCTs and all that? And we're like, you just use a linear regression to analyze the result of your test. You can put all these covariates in and blocking something you represent in there. So if you're doing an ANOVA on a pair Y, like two simple values of a category, it's similar to a T test. They're all linear regressions. And if I was doing this on an AB test, I could run a linear regression. Like I'm having this kind of like moment where Chelsea said like, it all.
- [27:36]
Chelsea Parlette Pelaridi
Goes back to regression. It's all, it's regress all the way down. Well, I don't know if this is too soon to bring this in, but I think what you're saying reminds me of something that I said when you reached out about this episode, which is sort of jokingly, but definitely not jokingly, that Cupid which people use for a B testing I'm pretty sure is just an ancova. Right? So the whole idea is that you take this like I believe I'm a little rusty on my Cupid, but you take kind of like pre test metrics that you have about the customers that you're testing on and use that to reduce the variance in the data because you're accounting for it, you're partitioning variance, it's blocking. I know. And you're just getting a better estimate, a more precise estimate because of all the variance that's out there, you're accounting for some of it that would have previously been attributed to random variants. You're now accounting for it with your category. And that's what an ANCOVA really is trying to teach you is like you can add these additional non experimental groupings or continuous variables and it' reduce the amount of variance in the error estimation, giving you a more precise, more statistically powered result.
- [28:50]
Julie Hoyer
And so this goes okay. It is all coming together. I am having like a mind blowing moment because that makes sense then where you want to use covariates that you know, explain like the outcome variable. So if you know age is a factor that would affect the outcome, that you're like trying to understand the variance for another category, like your campaign, you're like, well, this age and the next age group, we know they spend really differently. By adding that in as a covariate, you're helping, like, you're saying you are narrowing in on then being able to detect variants from your campaign because you've isolated and like muted the noise of variance from age that you know is a factor that affects it.
- [29:34]
Chelsea Parlette Pelaridi
Right, exactly. Like, whenever you add a covariate in a regression model, you're essentially saying, like, what can the other factors tell me after I have accounted for this variable? And so if age is really important in explaining how people are behaving, then you're basically saying, okay, if I know what campaign you got after accounting for all of the noise that happens because of your age, what does it tell me? And you're going to hopefully get a more accurate and precise measurement by including that. Now, you know, finding things to include that are actually useful can sometimes be a challenge, but if you can find them, they really help the precision of your estimates.
- [30:18]
Tim Wilson
My impression is that Cupid has become over the last couple of years, like, that is the, at least in the CRO world, like, oh, it's the latest kind of shiny bauble. You can reduce your runtime. You can, like, this is great, I guess what I'm doing. And there might be at least one person who's probably already been triggered because I know every time he sees Cupid, he winds up, I get text messages and he's essentially saying, but it's not magic. And I think it is probably because it's what you just got to, that you can't just assume that you're going to have covariates that you can identify that actually have an effect on the independent variable or the dependent variable. So you can't just like assume that you're gonna, you're not gonna have age, you know, within some cases, even if it is, you're not gonna have that data.
- [31:10]
Chelsea Parlette Pelaridi
And even more than that, I've seen some examples, again, not my area of expertise, but I've seen a lot of examples where you sort of have the same cold start problem you have in like, recommendation models where you may not have that data for, like a really important sector of the people that you're experimenting on especially this probably would come up most with new customers. And so people are. Right. Cupid would be so helpful. Right. It means you can run shorter tests, it means you can run smaller tests, it means you can have more precise estimates. But there's no free lunch. Right. You have to have this quality data that's going to behave in the way you think it will. And it's the same idea as an ANOVA or an ancova. Right? You're just, can we account for or partition out some variants that we sort of know is there is not the category of interest? Can we like section that off? And if you can, then I imagine Cupid is incredibly powerful. If you can't, maybe less so. But agree with whoever you're vaguely referring to. It's not magic and we shouldn't act like it is.
- [32:22]
Tim Wilson
He might be the same person that's had, we think, as many appearances on this show as you have. That's true. Name him by name. I'll definitely hear from him. So let me ask another question on those because I can think of in a simple website experience digital that there are things like what was the most recent traffic source? There are things like what device type are you on that? Both, if you're looking to a conversion, seem like they would be legit covariates. When you're talking about whether you're doing Ancova or whether it's Cupid, is that inherently a you that's part of the input to make your actual question of interest more useful as opposed to the flip side? Oh, we looked at the overall test results and now we're going to slice them by this other thing and see if, you know, significance pops up. Is that a fundamentally different thing where you're continuing to slice this is saying, no, I'm identifying this as a covariate so that my question I can get a tighter, better answer to my actual question. I'm trying to remove, to use that to remove variability.
- [33:48]
Chelsea Parlette Pelaridi
Yes, you're asking slightly a different question. Like if we're going to go in the linear model framework, you're asking a slightly different question when you say, is this relationship consistent between Android, iPhone, computer, whatever type users? That would actually be an interaction effect in your model where you're saying, does the relationship between my campaign and order value change for web based, phone based whatever. That would be an interaction term, which is something you can just add to a linear model, by the way, because it's so generalizable, which sometimes we don't realize with an anova. But that's a slightly different question than I'm just having campaign in here and I'm soaking up variants by telling you what platform someone was using. Because in that case you're just saying if I know your platform, can I like what additional information do I get from knowing about your campaign? Whereas the interaction specifically would model that relationship relationship differently for each platform and would allow you to answer that question of is it different? We probably look at the interaction terms there and see if they're significant. Or you could even use a mixed effect model for this type of thing where you say, oh, all of the effects are similar but they might deviate a little bit. How much do they deviate? You could answer the question that way.
- [35:18]
Julie Hoyer
As well because with an an ANOVA and covariates you're not actually interested in the difference between the covariates. Like you're saying you're just giving it extra information. But Tim was kind of posing it as more of a question of like finding out the differences across that extra covariate dimension of device type.
- [35:39]
Chelsea Parlette Pelaridi
Right? Yeah. So yeah, I mean it's just two different questions. I will say now I'm like having to rely on like really years old information that I haven't thought of. But I'm pretty sure for an ancova, one of the first things you're supposed to do, I don't think people do it and I might make you cut this if I'm incorrect. But I'm pretty sure one of the assumption checks for Nancova is that there's no significant interaction effect in terms of the covariate having different relationships, the interaction effects being significant. And I'm fairly certain that you're supposed to check that. And so in that case it would be like if you thought that was was happening, you wouldn't want to just include the covariate, you would want to include interaction effects because clearly they're meaningful. But that's a slightly different question that you're answering. So yeah, and I think that's a really good point about, you should be thoughtful about. Do you want to know that if you do include the interaction effects, it's not a traditional ancova, but because we're all brilliant and we know that this, you know, they're not discrete different models, it's just different forms of a linear model. We can so easily just like add an interaction effect and be like, okay, cool, like we want to answer this question, we'll add those interaction effects. And that's what I love about the linear model framework compared to the way that some people teach ANOVAs and COVAs like as separate tools that you can use.
- [37:10]
Julie Hoyer
And I think that's the hardest part, is because a lot of that thoughtfulness and the levers you can pull on these different statistical tools is how people think about them is really lost unless you deeply understand some of the math and the basics behind it. But as we know, a lot of times it's just a simple command in your code to run this thing. And if you aren't really good at checking all the assumptions and really thinking through the exact question you're answering, it's so easy to use a slightly wrong tool and get a number on the screen and think you're answering the right question and you're not. And I think that's what is scary in two ways. Like, you have to be really knowledgeable to answer that question of like, is this the right number to answer the business question I'm asking? And two, it's really easy for people to give you a number, and they haven't asked themselves that question or been thoughtful about it like that. Both of those equally scare me.
- [38:07]
Chelsea Parlette Pelaridi
And how are you supposed to be an expert? Both in, like, you need the business expertise to know what question is actually important. And like you said, you need the statistical expertise to know if the number you're getting is targeting that question and what the caveats there are. And that's one of the things that scares me the most, is like, how are you supposed to be an expert in both? I think the answer is you're not, and you have to collaborate.
- [38:32]
Michael Helbling
Yeah.
- [38:33]
Tim Wilson
Thanks, Julie.
- [38:34]
Michael Helbling
My anxiety had been going down as I was understanding this better, and now it just went right back up again.
- [38:39]
Tim Wilson
But I think there's the flip side. Maybe this is. This is part of the reasons that I wanted to talk about anova, because I have a very clear memory. And this was when I was sort of still learning R and I'd kind of gone down the. Okay, there's the benefit of just programmatically being able to do stuff that's not clicking around in an interface. And then I was trying to. I kept being told to learn R, you're just going to inherently learn statistics, which I don't really think is true, but at some point, I mean, it was just sort of said that if you're going to learn R, you're going to have to learn the statistics. They'll come hand in hand. And that didn't really happen. But when it comes to like, illustrating an anova, and I don't know if I've seen it since, I don't know what came first, I wound up arriving at a spot Where I said showing somebody who says, I want to have a deeper understanding. I don't know that this is full on a marketer, but it certainly could be an analyst and just showing distributions and saying if you're showing normal distributions with different variances and different means or the same variance in different means or showing two different examples of. Here's a case where like your example of 59 to $61 and $79 to $80, that's a really tight height distribution. Like, it does seem like you can visually help someone at least understand the nature of the variability so that when they go and interact with the statistician or the data scientist, there's a more productive conversation. Even if. And it probably also injects in like the same. The, the. The anxiety I've been living with now for seven years. I don't know anything. Like I'm. I can't. I can punch in and run the linear regression, but I am absolutely convinced that it's. There's something totally wrong with it. But. So I think there's a case where knowing. Developing some of the intuition without getting all the way to. I'm picking the right method and the interpretation of that correctly still has value. Where I get terrified is people just looking at a chart and not even having any, Any under any. Any intuition about why if they see $90, $80 and $70, they can't just make a declarative statement about the difference in those groups. So sorry, I just. I don't know why now was the time for me to mount my Trying to square that circle.
- [41:25]
Chelsea Parlette Pelaridi
Thank you for sharing. I think that's a valid fear. I think that fear hasn't gone away for me yet that like I'm doing something wrong. There's something that I'm not thinking about that makes it not ideal.
- [41:38]
Michael Helbling
Now really.
- [41:39]
Tim Wilson
I give up. It's time.
- [41:40]
Michael Helbling
I'm gonna go be a greener and maybe I should.
- [41:43]
Chelsea Parlette Pelaridi
There's probably lots of more qualified people that are like, oh, I'm past that. Chelsea's just not at that stage yet. But I do think it's helping PhD in statistics. Yeah, I mean, didn't help too much. It actually made it worse in some ways. I thought I knew so much about statistics back when I was learning the anova, and now I go, oh, I really know only a very little bit of statistics. But I will say that fear, I think, is a really good motivator to have the conversations we're having about the assumptions of an anova. What you're actually getting out of an anova. And I think that's really important. And I will say I had this thought when you were talking of we as statisticians or whoever it is who's putting out all this material on ANOVAs are not always good about talking about the real world applications of these tools. For instance, you may often hear with linear regression, with T tests, with ANOVAs, oh, it's robust to violations of this assumption. And that's true, but we don't really talk about that. Well, and I think it can lead to this thing where it's like, okay, technically there's an assumption of normality for T tests and for an anova, but we don't really talk about, okay, what happens when you violate it. And that usually ends up going one of two ways, which is people care way too much about that assumption and they're like, oh no, my Kolmogorov Smirnob test is insignificant or whatever, and they care way too much about it. When it is robust, the inferences you make are, Rob, they go the opposite. And people go, oh, it's robust, I don't care about it. And you go, no, no, no, no, no. It's robust to like minor violations of this. And so I think it does make the waters really muddy. Like if you were trying to decide, like, am I going to use a T test, Am I going to use an anova? Am I going to use a non parametric method to analyze my four group experiment model that I did, it makes it really hard to figure out what should you actually do because it's not always clearly communicated what the pitfalls are. So to validate your fears, you should be fearful. But also, people aren't really doing what they could do to help make it easier.
- [44:04]
Michael Helbling
And with that, we probably need to start to wrap up. Wait, what were you gonna say?
- [44:09]
Chelsea Parlette Pelaridi
Julie, I have, I have so many questions you should ask them. I do have something to share that I should have shared at the top, which is, as you know, as I've talked about a million times, I got my start in psychology. So while I don't use ANOVA much in my day to day life, I have a soft spot because it was, you know, in the intro stats classes that made me fall in love with statistics. I love it so much. My dog is named after an anova. Her name is Nova. So she's a analysis of variants, I guess, which I think we got after.
- [44:41]
Tim Wilson
We'D stopped recording last time. So that's, that's Michael's fault that we didn't manage to insert that for you.
- [44:49]
Chelsea Parlette Pelaridi
So it was very apt for me to be the guest here because cuz I love it so much.
- [44:55]
Julie Hoyer
Are you actually going to let me ask a last question, Michael?
- [44:58]
Michael Helbling
Well, I've got a lot of noise happening on my end, so yeah, go ahead.
- [45:04]
Julie Hoyer
I, I just wanted to. And this is probably a little bit of a can of worms to be ending on.
- [45:09]
Tim Wilson
But you know what, Mo is not here so you are just dishonor and.
- [45:15]
Julie Hoyer
Carry the baton for Mo. We talked a lot about like covariates, which means it would be an encove and but then you talked about like using an anova. Understanding the question that it's actually answering is that, you know, variation is explainable by the category you chose somewhere across these categories or across the category. And then you said you can follow it up with like responsible post hoc analysis. And we never really talked about a little bit of like covariates or post hoc, like which way do you go? And just the way you were talking about like using ANOVAs in practice, do you tend to lean towards one of those options instead of just a pure anova?
- [46:01]
Chelsea Parlette Pelaridi
Yeah, well, in complete transparency, I do not use these a ton in my daily life, but when I have mostly back in my psych research days, it you don't. It's not an either or, it's a what question am I asking? Because when you do a post hoc test, what you're usually doing is something like, okay, I had four campaigns, I want to know which two are different or which ones are different. And so post hoc tests can help you answer that, but you still could have covariates in that that are soaking up that variance. So it's sort of a separate question of like, do I want to include covariates to partition that variance as ANOVAs are want to do or not or. And do I care about these pairwise comparisons? Like if I have more than two groups, do I care which ones are different? And honestly, I'm sure there are some out there, but I really struggle to think of a question where you'd be better to use the omnibus F test that there is some difference, ooh, somewhere in here versus most people have questions and most people are going to action on those post hoc. So I can't imagine many scenarios where you'd want to do some type of ANOVA or in the ANOVA family and not want to follow that up with post hoc tests. And some might argue you should just start with those post hoc tests and control your Family wise error rate. But in any case, I think it's a very important part of actually gaining actionable insight from the anova. Gotcha.
- [47:39]
Tim Wilson
That does seem like that's kind of the weird also back when I I was trying to get some intuition around it and I found myself going down the and then you'll need to do a post hoc and the Tukey post hoc is the most common and I feel like I wound up in the the same spot. Like if you're always going to post hoc just feels like you're like, ah, I did this thing and then I'll kind of do this other thing. You're like, well, if you're always going to do that other thing, it's somehow. It has this Latin phrase on it as though it's like this kind of incidental tack on but you're almost always gonna use. It does feel kind of weird.
- [48:15]
Julie Hoyer
Have to do the ANOVA before you.
- [48:17]
Chelsea Parlette Pelaridi
Do the oh my gosh. Now this is the can of worms. This is like what I was.
- [48:22]
Julie Hoyer
My short question was the can of worms.
- [48:24]
Chelsea Parlette Pelaridi
This is what I was taught is sort of like you do the omnibus test and if the omnibus test is significant, it tells you something's going on in there, so you throw it out. Where there's a look, there's fire. I don't know that that's widely agreed upon as the appropriate way to control your error rate. And for in fact, I might be wrong, but I feel like I've heard that might be overly conservative, especially if you're also correcting for your family wise error rate in your post hoc test. So I would say my current recommendation. Ask me next time we talk about ANOVAs. My current recommendation is be thoughtful about the post hoc comparisons you're doing. So if you don't need to do all 10 groups compared to all of the other groups, don't. And then use some type of like a Bonferroni correction, a said act correction, the Tukey HSD that Tim was talking about, and just correct for your family wise error rate. There's lots of arguments about what counts as a family and what you should correct for, but we'll save that for the next episode. I'm on.
- [49:27]
Michael Helbling
That's right. That's right.
- [49:29]
Tim Wilson
Analytics power hour after hours. Yeah, yeah, analytics hour plus listeners. They can get access to the all.
- [49:36]
Michael Helbling
Right, well, before we start to fully wrap up, we do want to go around share our last call, something that might be of interest. Chelsea, do you have something you'd like to share as a last Call I do.
- [49:46]
Chelsea Parlette Pelaridi
It's a little out there, but it does relate to statistics and machine learning. You may have seen the movie Project Hail Mary is coming out soon. Based on one of the books that I thought was one of the best books I read read years ago when I read it. It's by the Andy Ware, I think is how you say that from the Martian. And it is not only an excellent book and apparently might be an excellent movie with Ryan Gosling, if you're into that. But the reason I'm recommending it here that is sort of related to statistics is I actually read a section of this in my stats classes or my machine learning classes because they have this really beautiful scene. I don't want to give any spoilers because it is is quite a bit into the book where they're doing something. Sciency won't go into it and they have this beautiful explanation. If someone goes, did you use artificial intelligence to do this? And the person says, no, we have to be able to test it in thousands of ways and know exactly how it responds and why we can't do that with a neural network. And I thought that was just such a great explanation in the context of the book. You'll have to read the book of why machine learning and some of the black box methods can be a little tough to swallow for some people. So for both statistical and literary reasons, highly recommend both the book and the upcoming movie Project Hail Mary.
- [51:11]
Michael Helbling
Nice.
- [51:13]
Tim Wilson
All right. My sister gave me that book for Christmas a couple years ago, and I didn't realize it was that he wrote the Martian until like after I was like, I gotta read something else by this guy. I was like, oh, he also wrote the Martian.
- [51:26]
Michael Helbling
So nice. All right, Tim, what about you? What's your last call?
- [51:31]
Tim Wilson
So mine is a post. There are times where I feel like I'm going back to the same wells, but usually when Jason Packer writes something, it is entertaining and really thoughtful. And he, along with Juliana Jackson, wrote a post called the duality of ChatGPT. And. And the premise is kind of there's two sides on multiple dimensions around discussing AI. Like AI will write our code and do analyses for us, or AI produces slop and won't make our jobs easier. And he just gets kind of thoughtful and has hilarious references, slips in like a John Lennon reference that is actually just kind of making a joke of a list. But it's a good read where he walks around kind of the duality and tries to sort of square the circle in each case. So I'm kind of hooked on people who are not completely in the bag for AI and are also not completely anti AI. And his was. I did actually I physically grinned. I don't know that I laughed out loud, but I was, I was definitely smirking while reading it.
- [52:49]
Michael Helbling
All right, Julie, what about you? What do you got?
- [52:53]
Julie Hoyer
Mine is very off topic and just something I enjoy, not related to the industry literally at all. But I hope one of you listeners maybe are looking for this type of app and I hope you enjoy it as much as I have. It's called the Short Years. And I had such a fear when I was having my daughter a few years ago. I was like, how the heck do people work a full time job, have a child and keep up with a baby book? But I was also like, I want to remember these things. I want to have pictures, I want to do the baby book thing. So I was on the hunt for an answer to that problem. And the short years has been amazing. It's just an app on your phone and it can give you daily questions. And so as I would lay in bed at night, I could just, just go through and be like, oh, here are three questions for you lately about your kid. And you can upload photos, upload videos, and then as you finish chapters, they just mail them to you. You've bought and then you buy the book. But you don't even have to buy the book or pay for anything before you start entering photos and information. So you could just go along and be like, okay, I've really stuck with this. I'm six months in, I'm going to order the book now. They send you the chapters, you put them in the book and then you can even extend it to the toddler years, which I think I'm going to do. But again, I've just been able to like keep up in the app and then I can decide to purchase or not. And I also feel like then any subsequent kids you can keep up with this. So you don't have like the first child got it all and the next kids got nothing. Like, I feel like this could help. So if you have any anxiety about baby books, I really love the short years.
- [54:32]
Tim Wilson
That's just funny.
- [54:33]
Michael Helbling
I was literally going to be like, yeah, that second kid, kid. Not so many details.
- [54:38]
Tim Wilson
I mean, I had to go through as soon as you said like the short years, like the, the former D1 volleyball player, I was like, and what were the short years for you, Julie? Was that like 0 to 18 months? At which point so.
- [54:55]
Michael Helbling
Well, my last call is, you didn't let me.
- [54:58]
Julie Hoyer
He was going to ask.
- [54:59]
Michael Helbling
Michael came first. Please stop.
- [55:02]
Julie Hoyer
Michael, what's your last call?
- [55:04]
Michael Helbling
So glad you asked. Mine is also AI related because it seems like it's dominating everything we do. But Anthropic ran a little experiment recently with an AI agent that they put in charge of a little shop in their office, and they basically gave it instructions to try to buy things and sell things to the people that worked in the office that would help it make money. And then they wrote up the results. It did a terrible job. And it's kind of cute what it was trying to do and kind of funny, but it just goes to show you that the level of complexity we can achieve with AI agents is not quite ready to replace us all yet. But it's kind of a fun read. So it kind of dives into a little bit of the details of what the AI was trying to accomplish and things like that and where it went wrong and what it did right. And they're going to keep running that experiment. I think they're working with an AI safety company, me as well, on that project. So. Kind of interesting. Okay, Chelsea, who knew that a TikTok about Monte Carlo simulations would lead to all this? The one thing that we could say TikTok was good for way back in the day, but thank you so much. It's incredible. I don't know why this is, but statistics, statistical concepts will feel hard to grab onto for us mere mortals. And you're a very unique and special person, and I hope people recognize that all the time in the way that you're able to, like, bring those concepts to life. So I just really want to say thank you very much.
- [56:52]
Chelsea Parlette Pelaridi
Thank you. You're thanking me by continuing to have me back on over and over and over at your podcast.
- [56:59]
Michael Helbling
Yeah, we're gonna secretly. We're just gonna shift the whole show, like, over. Just be like, oh, yeah. Now here's the statistical power hour.
- [57:08]
Chelsea Parlette Pelaridi
Next episode, statistical significance and why you shouldn't ignore non statistically significant lift tests.
- [57:16]
Julie Hoyer
Ooh, that sounds so good. That sounds great, love.
- [57:21]
Michael Helbling
Give us 10 episodes.
- [57:22]
Tim Wilson
Look, I've been on a lot of times I'm gonna do tell you what you really need to talk about.
- [57:27]
Chelsea Parlette Pelaridi
Exactly. Listen, I just don't want to talk about it anymore.
- [57:32]
Michael Helbling
Yeah, no, that's totally fair. Well, and this whole show came about because our listeners wanted more topics like that. And as you're listening, maybe there are other things you're like, please bring back Chelsea to talk about this. Like, we'd love to Hear from you. So reach out, let let us know. And you can do that on our LinkedIn or on the Measure Slack chat group, or via email@contactlyticshour.IO. so we'd love to hear from you. Yeah, I mean, this is awesome. Really great. And I guess we're gonna wrap this up. And I think I speak for both of my co hosts. Whether it's an anova, a T test, or an ancova, keep analyzing.
- [58:15]
Unknown Host
Thanks for listening. Let's keep the conversation going with your comments, suggestions and questions. On Twitter @analyticshour, on the web at analyticshour.IO, our LinkedIn group and the MeasuredChat Slack group. Music for the podcast by Josh Crowhurst.
- [58:32]
Tim Wilson
Those smart guys wanted to fit in.
- [58:35]
Unknown Host
So they made up a term called analytics.
- [58:37]
Tim Wilson
Analytics don't work.
- [58:39]
Unknown Host
Do the analytics say, go for it no matter who's going for it. So if you and I were on the field, the analytics say, go for it. It's the stupid, stupidest, laziest, lamest thing I've ever heard. For reasoning in competition.
- [58:54]
Chelsea Parlette Pelaridi
I don't think you ever said the title anova, I hardly know yet.
- [58:58]
Michael Helbling
Yeah, the title is sort of like a thing we don't actually say. And sometimes guests have come on and then they just talk about something completely off the wall, and then we change the title title again, not off the.
- [59:14]
Tim Wilson
Wall, but, like, that's an option. What the.
- [59:17]
Chelsea Parlette Pelaridi
Yeah, I could have derailed it.
- [59:20]
Julie Hoyer
She's like, wait, I can go rogue and you'll just completely reconfigure so it looks intentional.
- [59:25]
Chelsea Parlette Pelaridi
Good to know.
- [59:26]
Julie Hoyer
Yes, we will.
- [59:27]
Michael Helbling
Absolutely. Basically, the process of the show, Chelsea, just so you understand, it is Tim comes up with the titles, and then I just say whatever I want in the intro. So.
- [59:37]
Julie Hoyer
And then Tim reconstruction configures the titles. If they don't match up.
- [59:39]
Chelsea Parlette Pelaridi
Yeah.
- [59:40]
Michael Helbling
And if they're totally different, like, I just go off into left field to be like, let's make the title. First off, there's a committee for this, Julie, so.
- [59:49]
Chelsea Parlette Pelaridi
No, just kidding.
- [59:53]
Tim Wilson
It's like a really poor implementation because it can't complete a sentence and kind of goes all over the place and doesn't make sense. Or it's like, no, that's like the best. That's amazing representation. That's right. It's like presentation.
- [60:03]
Michael Helbling
Yeah. Tim to a T rock flag.
- [60:12]
Tim Wilson
And it's linear models all the way down.
- [60:19]
Michael Helbling
That's right. I saved that one for you, Tim, because I had an inkling you might do that.
- [60:24]
Chelsea Parlette Pelaridi
That's all I had.
- [60:25]
Julie Hoyer
I love that choice.
- [60:27]
Chelsea Parlette Pelaridi
So good.