← The Analytics Power Hour
Loading summary
Transcript175 lines
- [00:06]
A
Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language.
- [00:14]
B
Hi, everyone. Welcome to the Analytics Power Hour. This is episode number 283, and it's the show where finally, finally we're going to answer the question, does size matter? I mean, data set size, that is. I felt like the explicit rating we have for this show made it safe for me to make that joke, but prudence suggests that maybe I shouldn't take it any farther than that. So for the past 15 years or so, the business and analytics worlds have been obsessed with big data. Collecting it, storing it, and deploying increasingly sophisticated models and techniques to glean value from it. Some people have even gone back to school to learn more about it. And we'll get to that in a little bit. Yet arguably, many businesses are awash in small data, small and mid sized businesses and many nonprofits. For instance, as podcast listener Barrett Smith put it way back in 2024 when he proposed this very topic, quote, what are the right analytic tools for small organizations to use on the data they have to make decisions? How do we as analysts help these organizations be as data focused as the big orgs? So what can we do with data measured more in kilobytes rather than terabytes or petabytes or exabytes or even yodabytes, Something a term I learned as I was prepping for this very show. It's fun to say, though, a yodabyte. Yodabyte, simply deriding it for its laughable teeny weeniness seems like a missed opportunity. So let's talk about it. So I'm joined for this little discussion about small data by my co hosts, Mo Kiss and Julie Hoyer. Welcome to both of you.
- [02:00]
C
Hey.
- [02:00]
D
Hey, Howdy.
- [02:01]
B
That's coming in strong with the enthusiasm, trying out the throwing it to two people at the same time. And who will try to polite the other one.
- [02:10]
D
Yeah, see if we fight for the mic.
- [02:12]
B
Yeah, I know neither one of you is talking over people like I am right now, as I am talking over you right now. Oh, good lord, we have a problem. But of course, we do love to bring on a guest who's put some thought into whatever topic we're covering. And that actually proved to be a little challenging given the nature of this topic. So when Barrett proposed this topic, we're like, that's a great topic. Who can we have on to talk about it outside of just us trying to riff on it? And so it just sat there for over a year with us pondering it. Until we saw that Joe Domoleski had written a Medium post titled how to be data driven in marketing, even if your small business doesn't have a lot of data. So we pretty much picked up the phone and reached right out to him. Joe is the owner of Country Fried Creative, which is a full service creative digital marketing agency serving the metro Atlanta area. And he regularly publishes pretty great content on Medium in the Marketing Data Science with Joe Domoleski publication. I've actually used this content as a last call at least once on the show. So welcome to the show, Joe.
- [03:18]
E
Well, thanks Tim and howdy y'. All. Hello from Atlanta.
- [03:22]
B
So you're a Southern native, as belied by your first words spoken on this podcast, I take it?
- [03:31]
E
Absolutely. I like to tell people I'm a Southern Pollock.
- [03:36]
B
Okay. It feels like the Southern has kind of taken over the, whatever the Polish lineage there is, I think so a little bit. We're excited to have you excited to have this discussion. Maybe we can sort of kick things off by Joe having you tell us kind of what prompted you to write an article digging pretty deeply into the challenges for businesses that don't have so called big data to work with.
- [04:05]
E
Absolutely. And before I jump into that, just want to say it's really great to be on this podcast. I first indirectly met Tim reading his book and Tim needs to tell the listening audience what my claim to fame is with your book.
- [04:29]
B
Oh, oh, I want to hear this.
- [04:31]
E
Found a mistake.
- [04:34]
B
Found a mistake in it.
- [04:35]
E
I was the first to find a mistake. That's how much I read it. And we won't say what the mistake is. But I'm also friends with his co author Joe Sutherland.
- [04:49]
B
I would suggest that anybody who would love to see if they could find the mistake, if you go to analyticstrw.com or Amazon.com or target.com or walmart.com and search for analytics the right way, you can get your own copy of the book and see if you too can find the error that Joe found.
- [05:05]
E
It's a great book though. All kidding aside, enjoy this podcast. I'm normally listening to this podcast, walking my dog at 5:30 in the morning. So getting, you know, educated and enjoying the all the great guests you've had. So it's an, it's an honor to be here. So what, what led me to, to writing about small data? You know, I've been a small business owner for Golly, 22 years. And as we'll talk about in the show, everything skews toward big data and big algorithms. And you Know what about the little guy, the little person who just doesn't have a lot of data. And quite honestly, in my blog, a lot of the things that I like to write about are topics that are either not covered a lot or they're not covered in a way that is kind of relevant to a small business. And as a small business myself, I also wrestle with the problem of small data.
- [06:10]
B
Do you have clients who are, I have strong thoughts on this. Do you have clients who are coming to you saying can you run machine learning algorithms? That they're thinking that they need you and your team to dive into their data and you're like, guys, your email list is like 150 people. That's not, not going to be workable. Like, are you running into that in a day to day basis or is it more controlled kind of by the agency saying you're using what you can to help serve the clients with whatever they already have?
- [06:47]
E
So we are so far down on the spectrum. Let me reframe that question. Tim and Julie and Mo. And in some cases I am literally doing battle on some very fundamental issues of I don't believe in marketing or marketing is the same thing as sales, or the only thing that matters is revenue or what's a dashboard? Which I know is a favorite topic of all of yours. So in many cases even getting somebody to understand what data is, is, is kind of fundamental. So. And I'm kind of late to the game. You know, when I started the business, we started as a web design company. And of course with 22 years of history, we've, we've seen, you know, things like social media, search engine optimization, email marketing, and all the things we think about with digital marketing come on their own. And it used to be that we could sell on sex appeal, honestly, oh, I need a new website, make it pretty. They didn't care about analytics I've seen in the last five years in the small business space. And there's a lot of different definitions of small businesses. Let's just say 10 million and lower in sales just so people can kind of get a picture. They're not thinking about these things. So in some cases I'm actually having to educate them on the fundamentals. So they, most of them don't even know what machine learning is. Tim.
- [08:33]
D
Okay, but so then do you see that the problem that they're facing is fundamentally there might be the first step, which is like they're not using data at all, or there's a fundamental misunderstanding or trust. But then do you also see I guess the next step, which is are they making the best decisions they can with what they've got? Is that kind of like the evolution that you have to go through?
- [08:57]
C
Yeah.
- [08:58]
E
You know, Mo, when you're talking to very small businesses, many of them don't have a marketing department or they have a one person marketing department or they're outsourcing it, which of course I'm happy with that. We'd love to be their marketing department. So I think the first step is really, and this is to analytics pros like all of you who are way on the other side of the analytics knowledge spectrum. Cliches like you can't manage what you can't measure. We've heard that zillions of times, but it's fundamental to even the smallest of organizations. And so, you know, I think for many small businesses it's that education part that yes, marketing can be measured, which sounds fundamental and self evident and yet so many small businesses don't even think about that.
- [09:57]
B
But this is maybe going to take a little bit of a turn because now I'm fascinated. Like take email marketing, like if, if, because small businesses are running on, they have limited, they don't have a marketing department, they don't have often dollars sloshing around that they can say directionally it has to be good enough. My experience with consultants who do analytics support or digital support for small businesses and a little bit of my own experience is they're being really, really, they're like, we only have, we're spending $4,000 a month and should we put that in, in Google, in Facebook or in an email marketing? Like, are they not coming, saying, we want to spend as little as possible with you and when you do something, if you do an email marketing campaign for me, tell me whether it worked or not. Like are they asking that question or are they just saying you're a line item?
- [10:59]
E
Yes, some are. And then others, you know, we've got this concept of, you know, a minimally viable product. Right. We've all heard of that. And a couple weeks, couple months ago, I don't know, they all run together. I wrote an article, I applied that and I called it minimally viable marketing. And so, you know what I tell people is there is a minimum level, it's kind of your basal metabolic rate, if you will, of just I need to have some type of presence. And that'll be a website and social media and some other things like that. And so layered on top of that, of course are all the different, you know, things one might do with marketing. What I've seen for a lot of executives, which may just be one person, the business owner or, or a small management team, is there really isn't awareness of the numbers they already have. You guys already know this, but a lot of small business. You mean you can tell how many people went to my website? I mean, you would be shocked. Maybe not that many people don't even function.
- [12:11]
B
I can give you a number. It's not.
- [12:12]
E
Yeah, they got rid of my link counter. Why did they get rid of link counters? And I don't know how many people go to my site, you know, that sort of thing. And so a lot of times it's just uncovering what they already have. You know, when you send that email, you're actually collecting data without even knowing. Oh really? So it's just getting our arms around that.
- [12:32]
D
So talk to us once. Once these companies kind of start down the path of the minimum minim. Oh geez. Minimal viable marketing.
- [12:44]
E
There you go.
- [12:45]
D
And they're starting to like have this small data set. What are some of the techniques that you work with them on so that they can use the data they've got to inform their decisions?
- [12:56]
E
Yeah, I think mo, once I can get a client past that initial barrier that, you know, okay, marketing is different than sales. Yes. I need to track certain things. Then we start to look at things related to the quality of the data, the volume of the data. And when I'm talking to, you know, many of and I'm still very much involved, I have 12 employees, which makes my company, you know, bigger than two people. But you know, I'm not a 50 or 100 person agency. We've kind of got a nice little niche. I still do a lot of the selling involved in our services. Rarely do I see data or analytics be the lead. Normally we're going to start a campaign and try to do some more awareness of that. Normally it is in response to a specific pain point where data's involved but the client's not aware of it. But once we engage a client, I think the first step for us is to educate the client on the data they already have and they don't even know it. And then we can start to look at, you know, the volume, the quality and some of the things that I talk about in the article. And I normally like to pick one thing to focus on to try to prove the point. It could be the email open rate if it's a nonprofit, a conversion rate if it is a social media manager. Maybe it's the reach Very fundamental things like that. And I think once there's recognition on the part of a client, hey, there is this thing called data, and we actually have some. Then we can start to kind of explore, you know, different issues related to that data.
- [15:02]
C
And can we actually go back? Because you started to touch on it, and I know you talk about it in the. Your article more in depth. So I wanted to chat about this idea of not enough data that you had covered. And you kind of rattled off just now like different categories of not having enough data, because it's interesting, I hadn't even thought of, like, when we were talking, obviously, like small data sets. I didn't even think of just this broader idea of not enough data. So when you are working with clients and they realize, oh, I have some data, even if they had a large volume of data entry, it sounds like, though, you run into a lot where they still don't have enough data to what we're used to at larger organizations.
- [15:53]
E
Exactly. Case in point, a large email list. In terms of subscribers, some of the people that we're working with might be 10,000 people, most of them. Most of our client base, 2,000, like our local chamber of commerce. I'm a former board member of the chamber, and, you know, they're very engaged. And so what we see is perhaps a higher open rate, but we have a lower overall. You know, I think I called it in the article an end. But just, just the number, the sheer, the sheer number. When you're. You're dealing with very small or we're working with a. A nonprofit that literally started up a year ago. They have a hundred people on their email list now. What do you do with that? Right.
- [16:48]
B
But do you. Is that the sort of thing that if you say you have a hundred and you can talk through the math of saying, imagine if you had 200 or 500 or a thousand. So does it go. Do you wind up sometimes having a discussion of, like, if marketing matters, then it's who you can reach matters. We know how many you can reach with an email. Maybe we should consider trying to grow your email list. And if we're going to grow the email list, then we're going to need to measure whether we're growing it and what the cost is or is that not really how the.
- [17:25]
E
Tim, you actually nailed it. And I had this very conversation last week with that nonprofit that just started up. They were focused on getting donations and other marketing goals, which are important. And I told the executive director, I said, probably the most important thing you need to do right now is grow that email list because the donors are going to come and go. Who cares if you have 5 likes on your Instagram post. But you need to build that marketing database. So even with small data and even before we try to figure out how to work with it, that is a finding in of itself. Okay, we need more people in that database. We can do better marketing. And so there is value there, even with that low volume data set.
- [18:18]
C
And then another angle though is to say, like when you're the other example that you brought up, you know, if their list is more like 2,000 people or 10,000 people, like that's not necessarily tiny sample size, depending on what you're trying to look at.
- [18:30]
E
But no, now we're starting to have some decent numbers. Yeah.
- [18:33]
C
So then though, you know, do you ever run into. They simply have, this is like their email address. I sent them an email. And then they have nothing else about them because I know, like, what can you do with just a few data points? Even if it's on 10,000 people?
- [18:49]
B
Which was one of the things when you said small data, I always think of like the number of rows. And Julie's getting to another point that I think the number of attributes.
- [18:57]
E
Yeah, yeah.
- [18:58]
B
Like you talked about in the article. I was like, oh, it could be small data because it's.
- [19:03]
E
Yeah. I mean, you know, if we're looking at a classic data frame, to use a python term there, you know, we're talking, you know, when we say small. Yeah. It can either be in terms of, you know, features. And you know, that brings up a good point too. I was talking to another client of ours and we were trying to determine the optimal layout for a contact form. Now if, if you're needing lots of features, then you, you want to ask everything, right? Fill this thing out. You know, if any of you have ever adop a dog. Okay. The Humane society is one of our clients here in our neck of the woods. I'm a dog lover, love dogs, have Loki, he's a therapy dog all over our social media. And I looked at the form to adopt the dog because we had to put this on the website. I mean, I think it was almost as much as a FAFSA form. So, you know, my kids are grown and gone. Half your audience is like, oh yeah, that fafsa, that's awful. The rest are like, I don't know what a FAFSA is. Oh, you will.
- [20:24]
B
That's financial. Yeah, he's pulling the American, American Financial aid.
- [20:30]
E
I had to stick one on on you, Mo, the financial aid form. I don't know what the essay stands for. Basically.
- [20:39]
D
Got it.
- [20:39]
E
When your kids go to college, you have to fill out a form to document all your revenue as a parent, and it is hideous. And as a small business owner, it's like an audit. I mean, it's worse than tax form. And you know, okay, so you fill this thing out and okay, now we've got 50 things we know about our target market. Right? You know, that's, that's too many. But hey, if I just have a name and an email, is that really enough? And so, yeah, sparse. We might not have enough rows, but we may have too much or too few columns. In most cases, it's too few.
- [21:19]
D
But then do you also find that the businesses are. Do they overemphasize decisions with not enough data? I'm just thinking of that form example of, you know, we either have two fields or we have 200. But we think it's working because people are filling it out. And like, tell me about how they critically evaluate that when it is such a small sample. Like, that's the thing that, I mean, I feel like I have the inverse problem where people assume that everything is significant because we have so much data.
- [21:56]
E
Here's what's interesting in working with small businesses and, you know, for those listeners that are consultants with a handful of clients, you know, this, this episode is going to resonate with you because you, you encounter these problems a lot. It's not talked about a lot. It's not taught about in schools. And so in many cases, mo we're dealing with. And I don't want to make small business owners or small enterprises sound like, you know, they fell off the back of the truck and they don't know what they're doing. Although, you know, I've been doing this for 22 years. I still don't know what I'm doing. I've been making it up as I go along somehow, somehow figuring it out. But, you know, in many cases, they don't know what they don't know. So they're literally, you know, go back to the, you know, okay, you're tracking this data. You actually have, you know, we need to look at this. Many of them don't know that, you know, three fours of the people are abandoning the form. So they're just looking at the forms that come in. Oh, this is Great. We had 20 people fill out the form last month. They have no idea that Maybe there was 200 that gave up on it. And so part of engaging a client, where they're at with this small data is to create awareness that here's the big picture of what's going on. And I would submit that even the absence of data is kind of a finding in and of itself. Right.
- [23:26]
D
What sounds tricky there is like they're making some pretty simple mistakes. That's like a really hard conversation to have around. Whether you're overreacting to small changes, whether you're not actually looking, say, at the data that you're not collecting because folks are dropping off or they're using the wrong metrics. How do you start to navigate that conversation without kind of sounding like a jerk?
- [23:53]
E
You know, it's not easy. And I would say based on some of the interactions you guys have had with some of your other guests and just what you all do for a living, right? And analytics, Whether I'm talking to a peer, a fellow small business owner who I can appeal to, owner to owner, and say, you know, look, I understand where you're at, let me help you. Here's what we do. We take our own medicine. Or maybe you're presenting. It's kind of like emperor's new clothes, right? You're talking to the big boss and they don't have any clothes on. And you're just trying to get them to have a fundamental grasp of something very basic. And so you do have to approach it delicately. One thing that I've tried to do, and I guess this is the best place to insert this than anywhere else, is, you know, I decided almost two years ago at the, you know, tender age of 57, to go back to graduate school. And I am currently a master of science and analytics student at Georgia Tech. Part of the reason I am doing that, number one, is so that I get educated not on the small data, but, you know, on all the, all the cool machine learning things. But the other aspect of it is to have that brand where when I'm speaking about analytics, I actually have some academic credibility. It's not just, hey, I've got gray hair, I've been doing this for a while, but I can actually legitimately look somebody in the eyes and say, look, I'm literally studying state of the art stuff here. You don't have to do all these things, but at a minimum, you need to be doing X, Y and Z. And so that really kind of served as the basis for writing that article. Because even in grad school, everything is large data set. And so it's kind of like, hey, what about me? What about the little guy?
- [26:04]
B
Well, it does. I mean, that is a. I've got a little bit of an ax to grind. I mean, you're articulating this as a.
- [26:09]
E
More.
- [26:12]
B
Extreme kind of lockout than I was thinking that starting. And maybe some of this is your experience. And I'm trying to. As I'm thinking through the anecdotal interactions I've had, I feel like it's more small businesses or small analytics consultancies that have gotten kind of enamored by either this is what my platform, my CRM or my digital analytics platform or whatever. I have my media platform. It's just telling me stuff. I don't really understand it, but it's giving me numbers that are telling me I should pump more money into it. Or they are kind of in a mode of where we have to have more data. We can't do anything because we don't have enough data. Because there's so much talk out there of, oh, you got to feed the model, you got to feed the beast, you've got to build the big data warehouse. Whereas. And you make this point in your article. And I mean, I would. Matt Gershoff has sort of made this point like, if you have no data, then you're making a decision with no data. If you have small data and you do the simplest, dumbest little line chart and draw a conclusion, you may make a big mistake. You may say, I'm not thinking about confounding or seasonality or something, but it seems like overall you're going to be in better shape using small data that's noisy and messy. And sure, the more you know, the better. The more you know and think about something like seasonality, that's better. But shouldn't there be an encouragement to say, start with what you have, have somebody who's holding your hand a little bit that's keeping you from over interpreting something. But it sounds like your experience has been more. They're not even asking that question, which I'm struggling with saying, but they're paying somebody 500 or $5,000 a month to do stuff and how are they feeling like there's any accountability that that's a worthwhile investment. If they're not looking at data, you know, don't.
- [28:55]
E
Right. They don't see the value in it. And I think by, you know, I often tell people, you know, bad breath is better than no breath. Right. But you know, never heard that before. All kinds of Southernisms. But let me add a corollary to that. But it can be so bad that it Actually knocks everybody out.
- [29:19]
B
Right.
- [29:20]
E
So there's this fine line. Right. And so when it comes to data. Yeah, start where you're at. And normally I never encounter a business leader who doesn't have some sense of financial numbers. And I think that this is a place to kind of differentiate. And so we literally had an accounting client once, and the phone call started up. The marketing manager brought us in and said, I need some help. And we need marketing. We're a services company. I'm like, great, we're a services company, too. And we'll set up a call with the CEO. And the CEO starts off the call. Joe, I think you're wasting your time. I don't believe in marketing. Literally, we started the phone call and I said, okay, we have nothing to talk about. And then the marketing managers, like, wait, wait, wait. And, you know, I think they were.
- [30:28]
B
Posturing that for inspiring an employee.
- [30:31]
E
Yeah. And, you know, we started to have the conversation, and once the foot was in the door, the truth kind of came out. Well, we got burned and, you know, blah, blah, blah, blah, blah, blah. And I started appealing to this person who was a CPA on a financial basis. I said, you know, I actually have an MBA in finance. I know financial statements. You want to talk about ratios or whatever, we can do similar sorts of things with your marketing. And it was like a light bulb went off. She was like, really? I don't. I didn't know what I was paying all this money for. I wasn't getting any results. I said, well, you're actually tracking the numbers. And so I think from a. A data literacy standpoint that, you know, money. Money talks. Right. And so even if you can get it on those terms, you know, and then come over to marketing, and it could be any other data. Right. I mean, that's. My background is marketing, but it could be operations. What I find, though, is that marketing, of all the areas of a business, I think many times that's the last to get the analytics treatment. Other areas of a business, typically inventory and operations and logistics and finance and those sorts of things, and even sales. But marketing sometimes in a small business is late to the analytics party.
- [31:55]
D
That's so interesting. And I can see how. I mean, I can see the fact that, I mean, particularly like operations and finance, to some degree, can't function without data. But, yeah, I don't know, I've experienced maybe the inverse where those areas are, like, I would say less mature, definitely established first, but less mature. That's a. Yeah, that's a really interesting perspective. And I can see how for small companies, that would be the case with a small business.
- [32:26]
E
A lot of times, Mo, their sphere of influence might be a town or a county or a suburban area or metropolitan area. And frankly, they can get by with projecting. You know, we won't make this into a marketing class, but I, you know, we'll toss out there for, you know, when I went to marketing school, we had five P's of marketing. I think they're currently teaching the kids four P's of marketing. I read an article somewhere, there's seven P's of marketing. I don't care how many P's there are, but, you know, let's just go with one of them.
- [33:05]
B
Product. I only know four. Yeah, okay.
- [33:07]
E
Yeah, well, yeah, and you know, promotion, let's just say it's promotion. A lot of times their ad spend is, is kind of low. They don't have to spend a lot to maintain an online presence in a, in a small town. And so, you know, so they have back to that minimally viable marketing. They've got a website and they've got social media, they've got email. They may or may not do, you know, Google pay per click. And maybe they do. And so there's, there's that recognition of that. And so that creates another small data problem. Right. Okay. We're thinking about advertising, but we've never done it before. Now we have zero data. Now what do we do?
- [33:50]
B
But they're not hearing again, maybe where I'm floating in Columbus. I know analysts who support coffee shops that have three locations. And I've got a cousin in Colorado who supports a lot of service plumbers and electricians. For years now, it's been kind of a local SEO and even a local search engine marketing because they're saying, why would somebody know, you know, Tim's plumbing? I need to. When they're searching for plumbing, they hit this. But that they did. I pop up, that I show up in their location. And my, my sense has been, in the case of my cousin, she's not an analytics person. She's kind of a web web design website, web dev shop. But she's also kind of the content marketer. And she's often saying, how do I use the data better? My plumber client wants leads and he's paying me for his digital presence. And what am I supposed to do? A lot of this feels like there's kind of a tween place where the agencies that are providing the service, they're not big enough to be staffing full time. Analysts and they may not be kind of snowflakes like you who say, well, I'm going to just go get another master's degree to learn about it. And so where does that. I feel like I'm seeing much more where there is a hunger and an interest, but a lot of kind of fear and uncertainty about where does this plug in. And then relies on Google's, whatever Google's current term for their local business thing is Google Business Center. Is that what it is? I don't know. I feel like that's been rebranded a few times. Yeah. That they just say, well, I'm just going to log into that and hope that insights emerge. And that feels wrong. I don't know.
- [36:04]
E
Well, what's interesting. And of course we've gotten what almost halfway through the podcast and not said those dreaded letters AI but you know, we'll go ahead and take the opportunity to stick that in here. You know, there is some thought Google had some pretty big announcements at their marketing conference a couple months ago about making advertising in particular a little bit more self service. So you know, with AI augmentation, you don't have to know anything about analytics. You go in there and tell it, you know, hey, I'm a plumber and this is my target audience and let it, let it do its thing. Yeah, I'll believe it when I see it. Because I mean, they've been saying that.
- [36:51]
B
About keywords for, you know, 10 years. You're right.
- [36:55]
E
Yeah. And it's just not, it's not there yet. And you know, any of you that, you know that are listening, that have ever had to set up an online ad campaign or whatever, the user interface is, you know, rather, rather dense and thick. But you know, to your point, Tim, you're, you're absolutely right. And, and that's one of the things we're trying to do with our firm is find that sweet spot at a small company. They've got that one person freelancer that's kind of advising them on stuff and they get much beyond that. And you know, now what do I do sort of thing versus a big company which may have. I, I got to tell this story. You know, I'm working with a client right now who's on the larger end of the spectrum, they're actually a national company and they are just headquartered down here. Now don't fall out of your chairs. What I'm about to tell you and those of you that are driving while listening, don't drive off the road runners. Don't run off the Path and into the bushes while you listen to the podcast. This is a national company. Somebody set up a dashboard and the marketing manager can't even get access to it. But the executives don't want to change the platform. And so you talk about, they have data. It's actually being displayed in a dashboard. They don't want to change the contract. So we may end up going in there and setting up something to help bring the data to light as kind of a side gig. So it's like, don't mess with what's here. But we're flying blind and we need something to steer, and I've never seen anything like it.
- [38:37]
B
Well, so maybe to shift gears a little, I feel like we're kind of laboring in the getting from zero to one data point. And maybe it would be useful to say, okay, let's go to where there is a small data set. And we've already talked about kind of two definitions of small. One number of rows, one number of columns. And I guess you can have a lot of columns that are sparsely populated. So you go through in your article a few different aspects of that. But what are. If an organization, there is a person, an entity that is saying, I have data. It is not a lot of data. Where should I be starting and what should I be cognizant of? Maybe let's not worry about the pitfalls so much first. But what can be done effectively with small data?
- [39:43]
E
Yeah, that's a great question. And back to the use what you have sort of, you know, mentality. Some is better than none, more is better than some. But if all you got is some, let's use that. And I think the first step in that process, we have two interns working with us right now. They are, what's cool is they are also fellow students at other universities. So they're my interns. But I can also say, hey, I'm a fellow student, and we'll put them on projects. Okay. And what do you think? Again, they are in master's degree programs and analytics. When we start showing them small business data sets, what do you think their first reaction is?
- [40:33]
B
We can't get statistical significance with this. We don't have.
- [40:36]
E
Yeah. Oh, really?
- [40:38]
D
Okay.
- [40:38]
E
Yeah. The initial reaction is, normally we can't work with that. And I'm, you know, it's kind of a teaching point to them. Oh, yes, you can. And you're going to have to, because it's all we got, you know, so let's figure out how to, you know, to make use of it and you know, in the article I outlined some different techniques and you know, one of the things that we, you know, we talk about and again, to somebody who's familiar with analytics, you know, some of these things are or, you know, are going to sound familiar. You know, I think the first thing, you know, let's call it gut instinct, let's call it, you know, naive forecasting, you know, basic benchmarks. I think this is where, you know, experience can help provide some clarity on what you're looking at and what you can do with it. Case in point, you know, okay, we had, you know, 50 email opens on the, on the last, you know, campaign. Well, Joe, that's just not statistically significant. We can't do anything with that or even the corporate client, hey, here's what we know and it's all that we know. And I said, well, you know, that's, that's kind of a starting point. You know, we know something, it's better than nothing. Now that may be noise that maybe, but you know, we can apply some common sense to it and leverage what we know to try to make the most of. I guess it's kind of like being in the kitchen, right? Being short of ingredients. You know, when in doubt, put more flour in there or put an egg in there, you know, that always makes, I don't know how that works, but my wife always figures out a way, you know, okay, you don't have to go to the store, we'll figure it. I think she learned it from her grandmother. We'll just use the ingredients we have sort of thing. And I think that's true with, you know, with data. Another thing that I like to do is just, you know, kind of aggregate data into, into chunks or clusters or groups sometimes that, that really provides some clarity instead of looking at one big blob. Let's see what's in common. You know, I'm, I'm a Gen Xer, so you know, maybe I fit into a certain category in the, in the marketing data and it's not on the same level as maybe a Gen Z or like my kids, like millennials. And so you can do good old fashioned classification even with a small data set and get more insight than just looking at spreading it too thin in a big blob. That's been very effective working with little data sets.
- [43:23]
B
And as you were talking, there is an upside to, to small data is that you can actually look at the data. Like if you're dealing with a million rows, you can do distributions and you can do some eda and try to get a handle on the data and maybe pull and look at some of it. If you have small data, you know what, you can look at every one of those leads that came in and see which were garbage versus which ones weren't. And I think that goes with the. You know, if the master's student says yes, but you may have selection bias because your form's too long. Good knowledge to know, but this is back to bad breath or no breath, I guess, you know, if, if you have the small data and if you have a small data set, if you have qualitative data, or even if you want to collect qualitative data, if it's a small amount, you don't have to come up with some fancy natural language processing to try to assess the sentiment. You can read 100 comments, you can read 10 comments a lot faster.
- [44:31]
E
That's right. In fact, ironically, I am taking Analysis of Unstructured Data this semester along with the simulation class, which I've done before. I've actually written on my blog about some sentiment analysis restaurant reviews. But you're absolutely right. When you have that small data, maybe, maybe 50 people opened your email, you can actually know who they are and reach out to each one of them and ask them more about did you find it useful. And that sort of thing. Can't do that if you're dealing with 100,000 email opens or not. You can't do it as easily. Yeah.
- [45:09]
D
And you also mentioned controlled micro experiments. Can you tell us a bit more about that?
- [45:14]
E
Yeah, that's a, a fancy. You know, I tried to jazz it up when I put in the article, but I, you know, I think, I think at the end of the day, you know, we were all kids once and I, I remember, I've always been somewhat of a science and math geek. You know, I, I used to just put, put things together, little, little experiments, right. Steal stuff from the kitchen and try not to blow up the house or hook up electrical parts and try not to, you know, short out. Although I think I once did stick a paperclip in a light socket. And you know, my dad was not happy about that, but it was a cool.
- [45:50]
B
That's what siblings are for, to be blamed for the.
- [45:52]
E
Yeah, and I have a, I have a younger brother, Dr. Chris Domalesk, who has a PhD in this stuff. So, you know, he's, he's a smart guy, but I am, I am the big brother. And don't you forget it. Chris, if you're listening, but you know, I think micro experiments Again, if you're. If you're a small business, you're agile, you're flexible, you don't have to have a steering committee. You don't have to have, you know, 20 approvals. Just try something out. You know, you can do it in a controlled manner and see what works. You know, whether it's a little AB test or just, you know, I like to use the term a bake off. Right. You know, here, here's a sample. You know, you test this and you test this and see which one you know, seems to work better. And it may not be scientific, but it. You do kind of pick preferences. This happens a lot, you know, on the design side of our agency, too. How do you measure a logo effectiveness? I mean, really, okay, there's been papers written about that, but the end of the day, you get a small committee. Do we like it or not? You know, what do you like about it? What do you don't like about it? And so I think in similar manner, you can do that with other more qualitative marketing things.
- [47:14]
D
So one of the things that I was listening back to an old episode, and it was funny because Tim's last call was a podcast that I love. It was about choiceology, and it was talking about natural experiments. And so I wonder if this comes up as well. So Katie Milkman's a genius, but the episode was essentially looking at past incidents. I know, like, an example might be like, I don't know, we couldn't. We didn't have $4,000 that month, so we just didn't do any marketing. And so you use it as like, quote, unquote, a natural experiment to see what would happen if you turned off marketing? It's like, have you. Have you found that, like, looking back over historical incidents is like something that you can derive, like, meaning and direction from?
- [48:02]
E
Absolutely. And, you know, I think this applies to small or large data sets. Right. You know, data. Data tells a story. You know, one of the interesting things, and I don't know about your. All of you, your respective organizations or companies, but we saw a massive uptick of business during COVID Why might that happen when. When. When. When most things were buckling down and cutting costs, and we don't know if we're going to be here tomorrow. At first blush, you know, you would think that, you know, that would impact everybody equally, but it didn't. And we, I mean, we saw almost a 30% increase in our billings to customers. Now, why might you think that, you know, that happened during COVID Well, my.
- [48:53]
C
Guess is if you're working with small local businesses, a lot of people, you know, the larger organizations, if I'm remembering correctly, had the supply chain issues or, you know, things shut down. They couldn't have the big warehouses. And I remember there was a big push of like, go buy local, order from a local business, help them out when, you know, people can't go into their stores. So I think there was like a little boost for a while, right, from that.
- [49:17]
E
There was, I think, Julie, that was part of it, I think another part of it. What did we all do now? I let my staff work from home. We have. We have two physical offices. But, you know, I let. I let them work from home. And they're all creatives and they like that. That keeps morale up. But everybody worked from home for the most part during COVID And if you were a small retailer, a restaurant, mom and pop shop or whatever, you were invisible if you weren't online. So everything pivoted. People aren't driving down the road to see your sign. They are now sitting at home and they're on Facebook or Instagram or whatever. And so they had to pivot their marketing to online. And it was more than an aberration. I mean, it was sustained over about two and a half years. And so, as you were suggesting, mo, you can look back and kind of tell the story. At first glance, you'd say everybody kind of had a down and like, no, it actually didn't. You know, people needed our services more than ever because restaurants were converting to pickup. Right. So, hey, we need you to update our website. And if our website isn't updated, we're invisible and, you know, we're going to do curbside pickup and, you know, those sorts of things. And so, yeah, you know, data tells a story. And even if it's a small data set, it is worth looking at that.
- [50:47]
B
But that does get to one of the pitfalls that I really liked that you had in the article about ignoring context and external factors. And this doesn't seem like, to me, it's not tied to necessarily small or large. I mean, the number of anecdotes I have of somebody just ignoring an externality and looking at the data and saying, this thing jumped way up or this thing plummeted. People were aware that Covid was going on. People were aware of tariff stuff. But there's other things that happen. Are they aware of what their competitor is doing? And are they aware that if you see, if your data looks very surprising, an analyst, the first thing would say it's probably not a miracle. We found the perfect marketing message and this has caught fire. Like something might be going on that I may or may not be able to influence. So this email campaign did wildly better than any of our past email campaigns. I need to stop and think about my business and the operating context first and see if I can think about it. And again, maybe there you have a. The nice thing about small data is you can say I can really go look at what they did. Did a bot get a hold of an email or something like that? But it does seem like there's the same challenge, scales down that does go to a little bit of a data fluency, that if it's easy to sort of over interpret the data, the statistician's reaction will be, well, you need more data. That's not necessarily the right answer. You just need to be thinking about what's generating the data and what might actually also be going on.
- [52:43]
E
Yeah. Toward the end of the article, I think I summarized some of these things that I commonly see that are small, that are mistakes with small data and they're not necessarily limited to small data. I think you're right, Tim. They apply that other things. You know, probably the number one thing that I see dealing with small data are people killing campaigns too soon. You know, it's the classic, we've been running this for two months. Why, why, why haven't we set sales records? We're going to kill it. And you know, we, we know it takes time for things to work. Looking at the context which we, we, we talked about, you know, another one is kind of the inverse of that, which is, you know, okay, we just made a great sale. It must be the marketing. As much as I would love to claim credit for that, okay, that's a single data point. We can't even draw a line with that. We can just look at it and be amazed. And can we at least have a couple more to kind of draw a thing? And then of course, people waiting for perfect data. And I think that's the trap many analysts and data scientists fall into is this, this mythical thing. So I have an intern right now that's working with us. His name's Sam. Great guy. I'm going to have him listen to this and I want to encourage him. He'll graduate at the end of the year. We're looking at something, some of your listeners probably heard of it. Maybe you guys, just a classic marketing mix model. We're looking at the Meridian model. It's an open source model that's put out there and the sample data set that comes with the GitHub package, the marketing spend is like $240 million. Now I don't have a client that even their annual revenues don't get anywhere close to that. And so we're looking at this and Sam was, I had him looking in the model and I had him present this internally to our staff. And I said we're going to run some of our client data through it and you know, okay, a marketing spend on, you know, per month, even for a medium sized client might be $5,000. So you know, 240 million, 5,000. You know, what do you do with that? Of course, Sam's initial reaction. I'm not sure we have enough data. Oh, we've got enough data. We're going to make the most of it.
- [55:21]
B
Wow. Well, on that note, uplifting as it is, but it's a good segue to. We're going to start heading towards wrap but before we do that, we're going to actually take a quick break with our friend Michael Kaminsky from Recast, which is an MMM and GeoLift platform, helping teams forecast accurately and make better decisions. Michael's been sharing bite sized marketing science lessons over the past months and the coming months to help you measure smarter. So I'm going to turn it over to you, Michael.
- [55:58]
F
The synthetic control method has been called the workhorse of causal inference. Synthetic controls are used to generate causal estimates in situations where large scale randomization isn't possible or is too expensive. When we're doing causal inference, we're always trying to compare some treatment effect to a counterfactual what would have happened without the time treatment. The synthetic control method is a method for creating a counterfactual. It can be used in experimental contexts where a researcher has intentionally manipulated some treatment or in quasi experimental context where a researcher is trying to evaluate the impact of some change that wasn't intentionally manipulated. The idea behind the synthetic control is simple. We want to identify control individuals whose pre treatment behavior most closely resembles those of our treated individuals. So the idea behind synthetic control is to create a weighted average of potential control individuals that best match the treated individuals before the treatment. In the case of geographies, you might imagine the best counterfactual control for Houston is a mix of Austin and New Orleans and Dallas and that that mix is a better counterfactual than any of those cities individually might be. There are lots of different methods for creating these synthetic controls and correctly estimating the uncertainty in the causal estimates can be quite tricky, but when you utilize in the right experimental context, synthetic controls can help practitioners run statistically powerful experiments even when large scale randomization isn't possible.
- [57:15]
B
All right, so if you enjoyed that mini lesson, Michael and the team at Recast have put together a library of marketing science content specifically for analytics Power Hour listeners. For everything from building media mix models in house to communicating uncertainty to your board. Head over to www.getrecast.comaph. that's www.getrecast.comapH. all right, that was a fun discussion, and I'm not sure if I'm coming away from it more energized or depressed about the prospects, but hopefully it's a call to arms to not dismiss small data from some ivory tower of, you know, your ivory tower canva mo with your canva, you probably do have yada bytes of data that you're working with. How do you guys do you actually know data scale? Do you talk Zeta Exa. Exo. Yottabytes. You ever asked, how much data do we have?
- [58:23]
C
I thought it was Yodabytes. Where did Yottabytes come from?
- [58:27]
B
Well, that's your assignment. Find out. The last thing I like to do on the show is go around the old virtual podcast bar and share a last call, something that might be of interest to our users of any shape or form. And Joe, you're our guest. Do you have a last call you'd like to share?
- [58:46]
E
Yeah. So I'll do this in two small parts. First of all, for folks out there listening to this awesome podcast, if you encounter small data, don't be discouraged. You actually, hopefully you listen to this and you came away with some ideas. You're not alone. Because sometimes I feel like I'm alone. Am I the only guy who's twiddling bits here? But when I'm not doing graduate school stuff or running a small business, I like to kind of head in the other direction and get my mind off of computers and machine learning and numbers. And I have been reading this very addictive book series, and as usual, I'm late to the party. It is called Dungeon Crawler Carl by Matt Henneman. It is awesome. I understand it's going to be made into a TV miniseries. The premise. How many. How many of you? You all know what Dungeons and Dragons is? Okay, picture. And this sounds ludicrous, but it's normally the basis of a great concept. Aliens take over the earth and turn it into a dungeon crawl game. That's a game show. And people are trapped in there and it's like a live D and D and it's being broadcast across the galaxy. And the lead character, Carl, his cat can talk and her name is Princess Donut and it is everywhere. Dungeon crawler Carl there's seven books. The guy Matt wrote it during COVID and self published it. A book publishers picked it up. So it's a great business success story. You can't put it down. It is hilarious.
- [60:37]
B
Is he writing more?
- [60:38]
E
He is still writing more.
- [60:41]
B
Okay.
- [60:42]
E
Dungeon crawler Carl Highly recommended.
- [60:44]
B
That is if Michael Helbling was on here. I am deeply curious to know.
- [60:52]
E
I know there's listeners that are shaking their head going, yes, of course.
- [60:56]
B
Yeah, awesome. That's on my reading list now. Julie, what's your last call?
- [61:04]
C
My last call does use the dreaded two letters of AI, but it was a recent article that came out from Cassie Kozakov, one of our faves.
- [61:15]
D
Stop it.
- [61:18]
C
Was this going to be yours?
- [61:19]
D
Mo no. But I did also read it yesterday and I really enjoyed it.
- [61:23]
C
Yeah, it was really good. And actually my favorite part. So the title of this one is the Alignment why Gen AI Metrics Spark Debate, Not Clarity. She has a lot of great subsections in there. And actually why I wanted to use it as a last call was the fact that me and Tim were actually going back and forth and discussing the article. And it was interesting that she makes so many strong points in there and a lot of good points. But in general, I feel like it's one of those articles that I want to go back and reread. And it sparked a lot of good debate between me and Tim where I just think if you're. If you're starting to have conversations about measuring the performance of AI, something we noticed throughout the article and why I want to reread it is like there's kind of two themes going. Measuring the efficiency, quote unquote, of like the model itself, the large language model, like the actual process behind it, whichever one you choose. But then there's the larger idea, which I was more honed in on and interested in, of the actual output, return on investment. Did it help you reach your business outcome? And obviously that's something we talk about so much on the show and AI Gen AI being another tool to help you get there. So it's a good read. It's got a lot of good points and it definitely gets you thinking.
- [62:46]
B
It just seems like it mixed those together where I'm like, no part of this is not hard.
- [62:55]
C
Tim's like, let me drag my soapbox over here because I thought that's what.
- [63:00]
B
I want to be, is out publicly saying that I thought maybe Cassie was a little off on something, you know, that's.
- [63:08]
E
Oh.
- [63:10]
D
But so I was going to talk about something completely different, and now I'm going to pivot.
- [63:16]
E
Oh, I'm going to pivot.
- [63:17]
C
You don't want to do the Tim and do both.
- [63:19]
D
No, the other one's like a three for. So I had a really interesting conversation with the CEO on the weekend, just randomly. And we were talking about like, AI.
- [63:30]
B
Look at that little flex. She's at a professional sports game, just hanging out with the CEO, as one does. Okay, carry on.
- [63:39]
D
Anyway, we were having a very interesting conversation about AI and creativity because obviously that's something that canva, where I work, gives a lot of thought to and fundamentally humans still being at the center of it. And I was talking about. I was talking about like, how I use AI for generating ideas and just how I feel that sometimes I'm getting diverse perspectives. And he did a game with me that completely stumped me. And I Now I'm like, does everyone know this? Like, is this a thing that is known or is this a thing that is not known? And I am super like. Anyway, so we sat there and he goes, there was him and he had his son with him. And he goes, open your AI tool of choice. Right, Claude chatgpt, Whatever. And we had different ones. And it goes, you had to give it a prompt to give you a number between 1 and 10. Has everyone heard of this? Everyone gets the same number, regardless of the tool. And then it goes, give me another number. And then you go, give me another number. And we wrote our prompts slightly differently because each person wrote it in their own language. And you got the same number, and then it said, give me another number and you get the same number. And I was just sitting there like brain exploding. Because I think in my mind, the way I'd been thinking a lot is that companies that are leaning in really heavily here are going to have this advantage. And I guess kind of where I'm landing now is maybe that advantage will plateau at some point if I'm thinking that I'm using some of these tools for more hypothesis generation and things like that. Like, maybe there's going to come a point where it plateaus. But it was just such an interesting exercise to go to that I hadn't. And I was like, is this a known. Known that everyone's aware of is. Anyway, can we all.
- [65:35]
B
Can we all do that right now? Because I've Got caught up. And I'm asking, all right, give me.
- [65:41]
D
A number between 1 and 10.
- [65:43]
B
But we can kind of frame it as long as it's asking clearly that question. I can put.
- [65:47]
E
Give me an integer so we all type that in. Give me a number between one and.
- [65:50]
B
But it can be something else. Give me an integer that falls between 1 and 10 inclusive or something like that.
- [65:55]
D
Sure, Tim, sure. I don't know.
- [65:57]
F
We would be having phrases in different ways.
- [66:00]
D
Experiments.
- [66:01]
B
I'm not trying to break this. Yeah, I'm curious. Are we ready? We all going to go?
- [66:05]
C
Wait, wait, wait. Oh, hang on. I'm struggling over here.
- [66:10]
B
Okay, okay.
- [66:11]
E
Yes.
- [66:12]
B
Oh my God. So Mose and Tim's and Joe, you got the same thing?
- [66:19]
E
Yeah, same number.
- [66:20]
D
Okay, and then give me another number, Julie.
- [66:22]
C
Seven. Yeah, I got seven.
- [66:24]
D
Sure.
- [66:24]
C
How about seven? That's how it answered.
- [66:30]
B
Well, that's. Oh, wait, what? I got four.
- [66:36]
C
I got three.
- [66:38]
E
I got three.
- [66:38]
D
Okay, so. So this is interesting. I got four as well, Tim. But I've done this three times now and every other time I've gotten three is the second number.
- [66:47]
B
Hmm.
- [66:48]
C
I even said my. The way I prompted it after 7 was cool. Can I have another?
- [66:53]
E
I tried to be very casual and.
- [66:56]
D
Then normally the third number is also the same. So the thing that's very interesting about this is like just especially with the intersectionality of creativity, like are we actually generating more creative ideas or.
- [67:12]
E
No.
- [67:13]
B
Well, just because isn't it all.
- [67:15]
C
Situation. You know what I mean? It's all like the statistics of what is most likely in the combinations of the words and the. I mean, I don't.
- [67:23]
B
I mean it's works speaking a little bit.
- [67:25]
D
But there is also a very interesting article about work slop that we can talk about next time on the show.
- [67:31]
B
Yeah.
- [67:32]
E
Oh, AI slop. Yeah, yeah. I have a. I have a whole.
- [67:36]
B
Rant that was recorded that I never put it up. Val said that I could or should, but I. It was like six minutes and I was like, but who is this for?
- [67:46]
D
So, Tim, after our live experiment on the show, over to you. What's your last call?
- [67:54]
B
So I'm going to go very simple. And this was a. Kind of learned about this podcast called Lost Women of Science. Heard about it through 99% invisible. And basically the premise is the hosts go back with people that you haven't, you know, that you haven't necessarily heard of and they kind of bring them up and kind of talk through them. So specifically the one I listened to was June Bacon Bursy, which was the weather expert who answered the $64,000 question. And it's just. It is a well done. I'm kind of hooked on the. Listened to a couple other episodes since then, but if you're into kind of history and people who dive deep and kind of go a little beyond a Wikipedia entry on somebody and they tend to have fascinating stories.
- [68:50]
C
Nice. I just put that on my list.
- [68:52]
D
There's also a Spanish version which is very cool too.
- [68:55]
B
That is true. Yes. They recorded in both. Yeah, I made the note to do it a while back and I'd forgotten that. Well, this has been a pretty interesting discussion. I think we all now have a bunch of reading and listening material from the last calls alone, so.
- [69:15]
E
Whew.
- [69:15]
B
All right. But Joe, thank you for coming on, taking a break from your running a company and.
- [69:23]
C
And being a student.
- [69:24]
B
And being a student.
- [69:26]
E
Well, thanks for sticking up for the. For the small people, the little people, the little data people.
- [69:33]
B
If you have enjoyed the show, please leave us a review or a rating on whatever platform you listen to. I think Joe mentioned before we were recording that he has a podcast sticker on his laptop and YouTube. I do a sticker. If you go to analyticshour IO, click on the global now fill out a little form. We'll happily send you a sticker or two or three. You can also reach out to us on LinkedIn, any one of us individually, or the company page, or reach out to us on the measure Slack. Or you can just send us a good old email at contactanalyticshour IO. So regardless if you are analyzing yodabytes. Yodabytes, not yottabytes. Okay? Yoda think Star wars, not Seinfeld. Yodabytes of data. Or literally dozens of rows of data. Or whether you're just trying to explain and define what data is to one of your clients. For Julie Hoyer and Mo Kiss, I just want to encourage you to keep analyzing.
- [70:48]
A
Thanks for listening. Let's keep the conversation going with your comments, suggestions and questions on Twitter @NalyticsHour, on the web at AnalyticsHour IO, our LinkedIn group, and the MeasuredChat Slack group. Music for the podcast by Josh Grohurst.
- [71:06]
E
Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don't work.
- [71:13]
A
Do the analytics say go for it no matter who's going for it. So if you and I were on the field, the analytics say go for it. It's the stupidest, laziest, lamest thing I've ever heard.
- [71:23]
B
For reasoning in competition jacket's perfect record. If you're really feeling.
- [71:29]
E
I'm not jinxing us. I'm not jinxing us. I'm happy that I'm Gamble. They've gotten better because I'm a student there now. So that's.
- [71:41]
B
I was gonna ask you to put in your address so we could send you a little something, but I'm pretty sure if my records are right. Is that right?
- [71:50]
D
Yeah.
- [71:50]
E
Because I have your face on my laptop.
- [71:53]
B
Yeah, that's. We will not send you anything else with my face on it.
- [71:58]
C
You don't hear that often.
- [72:00]
B
I know.
- [72:02]
E
Well, the quintessential analyst. You know, that was too. It's. It's right up there with the rest of my stickers.
- [72:08]
C
Oh, my gosh. Hel is going to be so happy to hear that. It's going to make. Hel is weak.
- [72:12]
B
I'm just going to try to do it all, see how it goes.
- [72:16]
E
Show me, Tim.
- [72:18]
C
Yeah.
- [72:19]
B
Just me. It's all about me. Rock Flag and Yodabite. I forgot to actually come up with something.
- [72:37]
D
It had to be that.