← Always Be Testing
Loading summary
Transcript65 lines
- [00:00]
A
Foreign.
- [00:07]
B
Hello, welcome to another episode of the Always Be Testing podcast. I'm your host, Ty degrange and I'm really excited to talk to John Toski today. John, how you doing, man?
- [00:16]
A
I'm great, Ty. How are you?
- [00:18]
B
I'm good. I feel like as we've discussed, we've been trying to get on the podcast chat. It's been a multi month, maybe multi year endeavor to try to make it happen. Here we are. We did it. So good to see you and have you on the pod.
- [00:33]
A
No, thank you very much. Yeah, we've known each other for a long time and I know you've been, you've been having this podcast for a long time. So I'm honored and thrilled to be here.
- [00:42]
B
Amazing. For those of you that don't know John, you're in for a treat. For those of you. Do you already know you're in for a treat having been in the business partner marketing for a very long time, 25 years, is that right?
- [00:55]
A
Yeah, unfortunately it's, it's, I think we think we've passed that.
- [00:59]
B
Yeah. Yeah, we're getting up there in age, so I'm, I'm not far behind you. He's been in marketing leadership roles at ebay Affirm Opera Software. Currently running your own consulting firm, which is super exciting. So lots of knowledge and learnings to share.
- [01:14]
A
No, very, very excited to chat with you today.
- [01:18]
B
When you, you've been on high performing teams throughout your career, how do you approach building and scaling high performing teams? What does that look like? How do you think about that?
- [01:28]
A
Yeah, well, I think particularly in global environments, it comes down mostly to diversity. And I don't think this is something that I figured out for a long time. But when you think about diversity, some of the stuff that comes to mind is what do I have both men and women? Do I have people from different backgrounds? But some of the stuff that really started striking me as my scope grew and my teams grew in size and breadth and location was having enough people in different locations having enough of a breadth of experience and levels in the team. Because not just one of the big values of that, of course, is you get different ideas and different backgrounds. But I think more than that, there's a longevity to how the team can perform and grow. And part of it is with location, if you have one or two people that are very much satellites from everybody else that's in a central location, it's very hard to make them feel like they're a part of the team and you have to take extra Steps to make sure that they're connected. And time zones make a big difference with that. Obviously what level those people are at, if they're very junior and they're off many time zones away, it's really hard culturally to keep them in the loop and keep them motivated and make them feel like they're a part of the. The second one is having just a really good stretch in the levels of the folks that are on the team. And this is actually something that happened when I was at ebay before I left was we ended up starting with a very broad team of some very young people, lower levels, middle, and then obviously it's a pyramid, theoretically and everybody grows and suddenly you have a log jam because people have stayed and you have good people and suddenly you don't have growth opportunities for folks. And so trying to manage that really well, where you always have new talent coming in, it's very difficult to not want to hire somebody who's very experienced. Job wrecks are really rare these days. You fought and fought to go get ahead and you want to get the biggest bang for your buck. But bringing in another person at a more senior level is just going to create a challenge for you. Whether it's in six months, 12 or 18 months, or that person wants another opportunity. So trying to always come back and bring in younger folks that need more experience, that have more. A bit of a Runway, I think is super important.
- [03:59]
B
When you had that experience where you witnessed sort of a log gym, can you speak to the size of the team and maybe some of the other factors there that created that.
- [04:08]
A
Yeah, well, it's happens, it's happened several times, Right. Whether the team is 10 people or 35 people or a hundred people, you have more opportunities. The bigger the team is to try to move people horizontally into different roles. Right. So if you have different functions, you can create some cross training opportunities. I've been in this discipline for 10 years. There's no place for me to go. Okay, let's put you over here where you can learn something and that'll actually make you more qualified for a job higher up in the future. On smaller teams, it's just super hard, right. Like there's. Within my team, there will not be an opportunity. Like we won't be hiring another senior person between me and you. And so how do I start working outside of my own organization to find people opportunities? And that's, that's a tough one. Right? Like, especially if somebody's really, really good, I have to be willing to set them free and advocate for them someplace else to keep them at the company. And I think it's always one, how do I keep you happy and on my team? Two, how do I keep you happy and within our company? And three, if you're going to leave the company, will you at least tell people that it was a really great place to work, that you were respected, that you learned things and they can at least become kind of a promoter for you because people aren't going to stay with, with your company forever. Right.
- [05:36]
B
So yeah, great points, great points. I love that. When you think about the concept of always be testing, how do you think about creating that test and learn culture and mindset in organizations and teams that you're building and working with?
- [05:50]
A
Yeah, well, I think the companies that it's in their culture and it's in their DNA, you can feel it when you, when you step in, you, everybody watching knows they're in a company that has a testing culture or not. And I think great if you're in one of those. And it's a lot easier to make the case for making those decisions which you talk about it every, every week here, about what decisions and strategies you want to have and for those that you're not. I think it's, it's very, very difficult because testing has cost and we all know testing has an outcome. Right. That's why we test to get a positive outcome and to get some sort of benefit short and long term on our business. But you have to take a step back oftentimes to take that step forward. But to always be testing. I think there's a couple things. One is don't waste your actions without planning on learning something or if you didn't, what did you learn after the fact and taking the time to do the post mortem, making sure that people, probably broader audience than you're thinking about is aware of those learnings. I think it's like one of the biggest sins of companies is there's so much testing going on and product manager, analytics team will run something, they'll take all this time to write up a report on the outcome and it won't get socialized beyond that team or it gets socialized upward to senior executives who may or may not read it if they have time when they might scan it. But making sure things go horizontally to other teams is super, super important.
- [07:32]
B
That socializing down the people and telling the story of why you did it and what happened and what were the takeaways and learnings and next steps is just sounds obvious in hindsight, but it's amazing how often it just doesn't get done as much as it should or as effectively as it should.
- [07:48]
A
And then longer term, you get into the cyclical thing of people come and go and you realize you're testing something that somebody tested three years ago. And maybe it's because the person who wants to test it now doesn't believe what they saw before. That can't possibly. Right. There must have been something wrong in the test setup or they don't know. Right. Which is. Which is more likely. So the second thing I think is really tying the testing, your testing agenda to your strategy and starting back with what is the question you're trying to answer in the test? And I think this is something that I've seen over and over again, that whoever's proposing the experiment isn't actually answering the right question that they want to, isn't getting the right answer that they want. And it's because they've asked the wrong question. And so you might have a question which is, what is an advertising A B test or something, right? Like, what is the right messaging I'm going to put in here? And ultimately the way they set up the test might get to an answer that's helpful, but it misses the bigger picture in all the variables that you potentially should be testing. And so I'm running a messaging test. I'm going to keep all the creative the same. I'm going to try to keep all the distribution the same. But maybe that wasn't actually the question that you should be asking, or maybe the significance of the test isn't big enough to ultimately impact where you're trying to get to. And I think that's a really big one where at a company I've been at not that far away, not that long ago, the testing was trying to get. It was of a company that was very old. And so you're trying to get small percentages of percentages increase, but we're trying to grow 40%. Like these are not going to add up to enough that you're testing the wrong thing.
- [09:36]
B
I just want to pause for a moment because I love that one. And through a lot of our learnings, we've found similar in working with clients and just generally through experience, that thinking too small, for lack of a better term, is really a big trap in the whole testing thinking and then setting up tests the right way. So I think you nailed that from our experience, from my view as well.
- [10:00]
A
And I think the last one is this idea of what you have to believe is true. For this to be right, there's, I think it's very easy to get caught down into the experiment set up and doing a power analysis and how do we get to stat sig? And you might not need to get to statsig. The real question is ultimately for something to be true, I have to believe this and what is that? And if you don't actually believe that's possible, or if your CEO doesn't believe that's possible, then coming in with your experiment results aren't going to matter. There's a story around it and the statsig piece is, God help us, how long it could take to get to STATSIG on experiments. And at a certain point can you just call it and say directionally this is good enough? And I don't think that probably happens as much as it needs to. And I think it's again, cultural with a lot of analytics teams of feeling like they can't make a Recommendation unless they're 100% of the way there.
- [11:03]
B
Yeah, that's a theme that's come up a lot in different ways for me too. I think that could be a juice. Probably get some spicy comments on the topic of not waiting until reaching Stat sig. But I think there's. I love the thoughts of like data, quantitative data and qualitative data and listening to customers and other variables that matter and the stakeholders that you referenced. It's really interesting because not to take put words in your mouth, but over reliance on the data only as the signal and not other data points that matter. That can get tricky. But I think there's some value in what you're saying there and I think it really resonates.
- [11:49]
A
No, And I think you brought up the point of data versus intuition and data always wins, right? Because especially intuition is typically one person. Maybe you can get some consensus around an intuition, but for me, a lot of it goes back into the assumptions that are being made within the test. And I think that's where more often than not in my career when something, when something just doesn't seem right or seems too good to be true and you have to decide where you're going to put your dollars eventually and fund what outcome is, really go back and chew it up, right? Like at some point somebody, probably somebody fairly junior has made an assumption someplace. And a lot of times those assumptions are inconsequential. Sometimes they're hugely consequent. There wasn't enough eyes on this incredibly complex formula at some point. But that one assumption that was not right can tragically change the outcome. Of the test.
- [12:59]
B
Yeah, I love that Thinking of how did we get here? Why are we testing this? Having some brief documentation to support the why and the hypotheses sounds elementary to a lot of people in experimentation and in growth, but is so important because it allows you to revisit those things and say, why are we here? Why are we doing this? Why are we going through all these pain points or efforts to. It just helps kind of ground you in that. I really like the thinking. It sort of reminds me of recent conference, great tech company, great partner, going through data and trying to determine like, what's what and kind of going down the direction of tracking. Right. And figuring out how are we tracking fully. And without giving out too many details, it was like, okay, we're seeing Data points of 0 and data points of 100. The old adage of like, if you see much as 100% or zero, you start to. Your intuition kind of peaks and you start to think, okay, is there something wrong with this data set? Is it actually tracking fully? So it kind of like reminds me of a way that a small subset of data or could lead to an intuition that leads you to get to the right data solution. So I'm sure you've seen examples of that.
- [14:12]
A
Maybe aside my eyes, it's the one I would bring up is, well, this is the average. It's like, well, what's the shape of the curve? Because if the shape of the curve is like this, the average doesn't matter all that much. Like, I want something good like this. So I have confidence that there's enough significance on that average point. Otherwise that average, it's not a number that matters for anything.
- [14:37]
B
Yeah, no, I love that. So good. How do you kind of balance the intuition piece, which we talked about, to deciding how to scale things and fund things further? When you have data sets, intuition, internal stakeholders, opinions, perhaps a test result or multiple test results, how do you kind of balance all of those. All of those things that you're ingesting to say we're going to do this again, we're going to scale it across the site or across different campaigns. How do you kind of balance all those things?
- [15:08]
A
Well, I think from my position, it goes back to what is that question we were trying to answer and does the outcome of this experiment answer it or not? And so a test and a bubble is interesting. How it relates or is relative to the business around it, I think is super important. But more importantly, I think, I think people generally undervalue their relationship with their finance teams and Senior executives. Ultimately every organization I've been in finance. Finance runs things right. And not just in how we're counting things, but going in and looking at those assumptions themselves. Back to the point of did somebody make a poor assumption? Your organization might be working with a set of numbers for their assumptions that are generally accepted within your organization. You go up a level and over to finance, they're working with different numbers. You know, they have a different CAC or a different LTV that they believe in or their understanding of your margin is different because they're bringing in some numbers that you're not being held to. But when they go look at your experiment in your numbers, they'll say, well, what is this number? This is wrong. It's like, oh, this is the number we use. You're like, no, that's not the real number. It's like, wait a second, there's a real number. And I think that's being able to storytell to. To who holds the purse strings and build relationships so that you have credibility with that organization and the, and the personal relationships. Somebody day to day you're working with and then helping them, helping you to have credibility up towards whoever it is, the cfo, the CEO, particularly if you're looking for a decent amount of money.
- [16:53]
B
Yeah. So critical, so important. Spot on. Looking ahead to what's coming. What trends do brands need to really be thoughtful about what's coming for affiliate. For performance marketing. Looking ahead to the next two or three years.
- [17:06]
A
Oh man, it's. It is. I think if there's. We said 25 years before, I think if there's been one constant, it's that you don't know what's going to come next. But something always comes and I don't know what it's going to be. Right. Like I would say two or three years ago, I don't think anybody, maybe three or four really saw Buy now pay later as a massive driver of performance marketing traffic. And obviously I enjoyed that with my time at a firm I in the last couple of years and understanding there's a critical mass of people who are at the right point in the funnel where they're pretty ready to buy, but they can still be encouraged of where to buy that out or what to buy. But they're low enough in the funnel and I think is that AI. I don't know. I don't know that that's going to happen quickly enough. I know we've been talking about creator and influencer for ever about like this is where it's all going to head. I think we saw a lot of the sausage being made with the honey lawsuit creators realizing, I don't know that I actually fit within this world. I think we're still going to see some backlash on that. And how again if marketers brands need to separate out their lower funnel affiliate programs with their upper funnel branding and creator because there's lots of traffic but it doesn't convert. And so is it fit with an affiliate, is it truly performance marketing or is it more brand marketing? And I think everybody's got their own, their own feelings about that.
- [18:51]
B
Yes.
- [18:52]
A
Yeah.
- [18:53]
B
Another one we can put in the comments for a spicy chat.
- [18:57]
A
And retail media I think is the other one that if you're at a big company like Walmart or Target or Home Depot, it's probably been a swell that's been happening for years. But I think for a lot of us who are not at those companies, it kind of came out of nowhere A little bit is affiliate and other types of performance marketing and extension of the retail media network. And from a investor point of view or executive point of view, it makes sense because that's the story right now. You're going out to the market with, you're building media network, it's driving your revenue bottom line. And this is additional inventory effectively that you can put in there. And we've seen companies reorganizing their performance marketing within retail media but it doesn't mean anything's really changed. Now the question is, are these brands able to actually use their first party data effectively to third party sites? And I think that if somebody can solve that, that would be huge. But those are the same problems we've had for years of give me access, give me control over the space on your site and I'll use my data to target something really good that will convert well for you. And being that publisher saying no, I don't want you to have control over what goes there. I get the control. There's a disconnect there.
- [20:24]
B
Absolutely. Yeah. And the retail media network is such an interesting one. It's like anytime you give media buyers and brands that the power of improved data or a closed ecosystem and it's like it brings up debates of like what's happening to the open web and third party privacy debates. And going back to your other point, there's a lot of great movements that you highlighted but I definitely find it fascinating the kind of redefinition of appropriate attribution and thinking through how do you value partners, influencers, affiliates, performance marketing related or not, it's something I think we're both kind of wrangling with and trying to make sense of and seen as an issue. And I think I love helping. I think you and I both, it's like there's ways to better measure. It's just shocking how often it's not measured effectively or even thought of effectively or discounted or not. So that's a, that's a hot button, one that we could probably have a whole episode on.
- [21:30]
A
No, absolutely. And it's so cyclical and whether it's Nike now pulling back, I assume they still are pulling back from bottom funnel spend, put more money up in brand because they shifted too hard or airbnb Brian saying years ago everything goes in brand and now they're entered back into performance marketing over the last couple of years and have been quietly putting dollars back there. There's no right answer, but it's really challenging if you're a CFO or a CMO trying to understand how it fits together and look at things apples to apples. Because they're not ultimately and everybody wants a one to one explanation of everything. And I mean we've been trying to do this for a long time.
- [22:20]
B
Yeah. Yeah, absolutely. Speaking of coming waves and trends, obviously AI is top of mind exploding, clickbait, good, real everything. What's got you kind of worried about it and all the rapid adoption. What's got you excited about what's coming for marketing and E commerce?
- [22:37]
A
Yeah. Well, it's both exciting and terrifying about AI is a marketing channel. Both the technology that people are trying to build or thinking about building that the big AI companies have their own plans that they're not telling anybody. I assume they've thought about it but are not really telling anybody about. And so you have tons of venture capital that's been ready to explode for years now that has found its outlet within AI. And I just feel like there's going to be an awful lot of money wasted while this is all getting figured out. And it's been very interesting consulting for the last several months and talking to lots of different businesses on various sides and the steps they're taking. The smart ones are figuring out all the possible outcomes and then putting together plans and acting on those plans now so they don't get caught flat footed. People are just sitting back and waiting to see what will happen. So that's, it's exciting and scary. What really worries me though, I think, and this is partially because I'm getting old and partially because I have a teenager, but I think there's this skills gap between these, I'll say mostly college students now. I mean we've probably, most of us have seen some of the, the graphs that have come out now that school is back in session, it's late September here now. And see the ChatGPT queries dive at the end of the school year and pop back up. Right. So like the raw amount of interaction that's happening within the platform is being driven by a relatively small group of people. And culturally there's just such a huge difference. And as a, as a manager and a mentor trying to really understand the thinking that goes behind how a younger person uses AI versus how I, an older person uses AI, the short answer is probably much less effectively and I'm using it much more consultatively or maybe just trying to get some basic ideas and then I'm going back and trying to put together some thoughts and I'll maybe go back a little bit later. And how does this work versus just dumping everything in there? And so again, culturally that's interesting. From a business ownership standpoint, most businesses don't have their own internal AI models today. Different businesses are more willing to dump information into AI about their businesses. And I think right now you have this conflict of everybody in the company should be using AI. God, don't put that in. So how do I use this as a tool? And how do you then train this dichotomy of older employees and younger employees so that you have policies and language that support both? Or does somebody effectively get left out in the cold or there major gaps in how people communicate? If I have somebody working for me that's using AI for most of their work and I don't understand that or they don't understand the downsides. Like I think there's a big culture shock that's going to hit us here in the next few years.
- [26:03]
B
Yeah, those are real. I'm glad you're talking about some of the possible negatives. And it gets me excited about the educational opportunity to bring, bring awareness and how to bring it to good and how to leverage it just like we have previous technologies. So my hope is that as the optimist I can be at times we can educate the heck out of folks and support it and enable people to create some good from it. But I think those questions are real and it's definitely got its upside and downside currently and in the future. So spot on.
- [26:38]
A
It's not new, right Ty? I mean this goes back to computers in the workplace, this goes back to the Internet like generations have dealt with Technological changes.
- [26:49]
B
Yeah.
- [26:49]
A
And again, maybe I'm silly, but it just seems like such a big step to. It's not access to information. It's not access to something that can accelerate your own output. It's potentially access. It's generative. Like the Internet's not generative. You know, personal computing wasn't generative.
- [27:13]
B
You're handing controls and output an agency to something else that you don't necessarily control.
- [27:19]
A
Yeah, it's wild. I'm excited and scared.
- [27:22]
B
I mean, that's an amazingly good summation. Speaking of software, we're going to segue coming down the home stretch, drop some awesome knowledge. Just great stuff. Give us a little insight into John, what's the software, since we're nerding out a little bit that you like, that you recommend that you want people to check out.
- [27:47]
A
There's a lot of software I use a lot, but I don't love it. I think the app that I love the most is. I think it's done by Cornell. It's called Merlin Bird id.
- [27:57]
B
Oh, my God. I love it.
- [27:59]
A
Yeah. Basically, if you hear a bird, you open it up and it listens and then pretty regularly will tell you exactly what kind of bird that is.
- [28:08]
B
It's epic. I love that app. My wife turned me onto it. It's so cool.
- [28:13]
A
Yeah. It probably has again, to do with being an older guy that's more interested in what bird is at outside than other things. But I know now that I have a house bench, building a nest in my apple tree.
- [28:24]
B
Love it. Beautiful. What about a consumer product that you have to tout?
- [28:32]
A
This might be silly, but I really love my floss. And it's made by a company called Dr. Tongue T U N G and they make a smart floss. And I was not growing up, I was not a flosser like a lot of people and gonna say, you're so responsible. Right. Well, in it, you know, I started using like the Glide floss. And like, that was a lot better and it was smooth. And then it's like lots of microplastics just like shoving into your mouth, like rubbing. But this doctor Tongues is. It's like stretchy. I can't describe it, but you use it and it actually feels like it's working. Is the best way interesting if regular floss feels like you're jamming something in there and maybe something comes out. The doctor Tung's floss feels like it's actively pulling things away.
- [29:25]
B
I love it. Very cool. Love to put it toski at checkout in the show. Notes. I love it. What's something that people might not know about you?
- [29:35]
A
I love to cook and as I've gotten older I love to cook more. And I have wife and I mentioned a teenage son and I basically shop for and cook all of our meals. And I don't. I actually shop. I will go pick out my own produce and work the aisle. I'm not an instacart guy. I probably cook dinner like five nights a week.
- [29:59]
B
That's amazing. That is so cool.
- [30:01]
A
Yeah. And it's something that I used to feel like was another chore and I think I've gotten more zen about it. And the food tastes better when I'm like, this is actually my hobby, not just another thing that I have to do.
- [30:13]
B
Oh my gosh. Are there any dishes that you are go to's or favorites?
- [30:19]
A
Well, I'll add in that I'm vegan, which usually either makes people roll their eyes or makes them not interested in hearing what. What dishes I would. Would recommend. I'll leave it there.
- [30:32]
B
That's amazing. Okay. Farmer's market. I. There, there's. Having lived in near your area, I may have some. Do you have any spots that you like to go in particular or anything that you just love to or is it just straight up grocery store?
- [30:48]
A
I will go to like four grocery stores to like get different things at different grocery stores. Not all the same trip oftentimes, but you know how just some places have the right stuff and some places don't have the right stuff. So yeah, I'm. I'm willing to take the time to get the things that I.
- [31:05]
B
Okay. Hot for non vegans out there. Is there a vegan dish that they should try that they won't squirm at that might get them interested in either your cooking or. Or eating a vegan meal?
- [31:16]
A
Soups are usually a good place to start because within like a lot of people you talk about lentils and pulses and they're like, okay, that's very vegan. But within a soup like lentils and beans and stuff is usually more palatable for people. So yeah, like a good hearty soup. I remember somebody told me years ago, they're like, sometimes I just have a pot of soup for dinner. I'm like, really? And two nights ago I just made a pot of soup for dinner.
- [31:43]
B
And yeah, that's awesome and inspiring me and hopefully the audience out there. That's commendable and I think it's so cool. And it's like a. It's like a. Just a good thing that's amazing. John. I so appreciate you jumping on. We talked a lot of great learnings and it's always good to talk to you. We always enjoy our chats. And for folks that want to connect with you and learn more, what's the best way for them to track you down?
- [32:07]
A
Yeah, you can email me anytime. I'm at toskymail. T o s k e y mail.com toschi.com is not an active website, but it will take you to my LinkedIn page.
- [32:19]
B
Ooh, I like that.
- [32:21]
A
Yeah. I've been threatening to build a website for two decades now.
- [32:25]
B
It might have your recipes one day, maybe not today. Secret Secret. Thank you man. John, have an amazing day. And thanks for joining me.
- [32:35]
A
Thanks, Ty. I really appreciate it.