← Always Be Testing
Always Be Testing
#112 Affiliate Networks Drop Honey: What Really Happened | Ben Edelman & Tye DeGrange
00:28:50
Loading summary
Transcript33 lines
- [00:00]
A
Foreign.
- [00:07]
B
Welcome to another episode of the Always Be Testing podcast. This is an emergency episode and a very important topic that's hit the affiliate industry around Honey and downloadable toolbars and the significant issues that we're seeing with them in our industry and brought on none other than Ben Edelman. He, he is a second time guest, someone I've had the pleasure to work and collaborate with for about what, 15 years, collaborating now and catching fraud, assessing the quality and he's going to give us his perspective on what's happened with the honey. Issues that we're seeing pop up significantly with networks removing them from their platforms. Ben, nice to see you sir.
- [00:50]
A
Great to talk, love the subject.
- [00:53]
B
Happy New Year. We're getting right into it and maybe you can start off with the what we can kind of go through the facts of what you've witnessed with Honey and understand for folks what's happened from your perspective.
- [01:07]
A
So let's just start at the beginning. Obviously a lot of your listeners are experts, but from first principles, affiliate marketing is supposed to reward the affiliates who actually, quote unquote, caused the sale. What does that mean? In a world of some piece of software on a user's computer that can pop up and say something, do something, when the user is already at a merchant's website. In particular, if an affiliate drives a user to a merchant's website and then a shopping plugin like Honey pops up some message, who should get paid? Well, this is not a new question. It was addressed 20 plus years ago through what became sort of a CoC like code of conduct. And the prevailing understanding is a principle called stand down. If someone else has already referred through an affiliate program, this user, to this merchant, then the software on the user's computer has to stand down, has to do nothing in order to give that first referral a fair chance to get paid for the work that they did, that's not the only way it could be handled. You could imagine a different rule that just says, hey, this idea of putting software on a user's computer, that's no good, that's not incremental for merchants. We don't allow it at all. Personally, I'd be super resettled to that as a rule, actually. But that's not the rule that people agreed on. This is the rule. We've got the stand down rule. And the problem is stand down has been notoriously finicky. Sometimes in good faith. A software program that's trying to stand down in the right circumstances doesn't. Maybe it had trouble noticing a new format of Link or a redirect, something like that. It's always been suspicious. The number of mistakes has been kind of odd. The mistakes are always in the direction that provides a financial benefit to the software publisher and takes money away from the normal we publisher. So you can't help but have a suspicion. What happened with Honey was different though. The nature of the finding, the nature of the violation left no doubt about intent.
- [03:01]
B
Yeah, absolutely. And just to give you my quick perspective on what, what's happened. So Affiliate Summit was just recently, literally the big kickoff of Affiliate Summit rack and announced they're removing Honey from their network. Impact followed suit. The next network, Awin has taken action as well in violation of their terms and essentially they've. And you're going to give your take on it, but essentially Honey has structured their code to evade detection from affiliate networks and affiliate managers, correct?
- [03:34]
A
That's exactly right. So there's a bunch of people in the world, including network quality teams at the networks, some merchants and then some independent people like me who do it either, frankly as I don't know, a hobby just for fun, or sometimes do it on a paid basis. Networks or merchants could contract people like me. Historically I did a bunch of that, these days a little bit less. But we look for violations. So any shopping plugin that wanted to break the stand down rules in the very convenient way I mentioned a few minutes ago would immediately confront the problem that they might get caught. So how can you break the rules and not get caught? Well, we see in Honey's code and in its config files and in its telemetry and in hands on testing all showing the same thing, we see that they have ways of distinguishing people they think are probably testers from people they think are pretty likely to be safe. For example, if you have a brand new account, first 30 days, say, well, that could be a tester. Obviously lots of normal users also have new accounts. Everyone's new in the first 30 days, but we're not going to break this rule for the first 30 days. So any tester who creates an account and tests the same day or the next day or the next week, those are all within the first 30 days and they'll never get the violation. Actually, it's so much worse than that. The really terrible one that left zero doubt about what Honey wanted to achieve was checking the cookie jar. Cookie jar is the place where your browser stores cookies. If you have logged into a dashboard to see how much money you made as a publisher, or if you're administering a program to see who your top affiliates are, how much you're paying them, who has what conversion rate, that kind of thing. If you've logged into a dashboard from any of the big networks, Honey can actually read those cookies. Remember, it's a browser plugin that has read all permissions. They can read data on all sites that you go to, including reading the cookie data. Man, who thought they were signing up for that. So if they see that you logged into the CJ dashboard or logged into the impact dashboard and so on down the list, say, oh you're an affiliate marketing pro. Well, we're going to follow the rules for you. And meanwhile for normal users who haven't logged in, now that wouldn't catch all testers. I never do testing on a machine that I use for my personal activities. I have a separate test environment, multiple test machines. But anyway it would catch a bunch of people and it's really not right. It shows what they are trying to achieve and the fact that they are trying to achieve it is the violation they shouldn't be hiding from testers.
- [05:52]
B
For those following at home, multiple years of browser extension debates, conversations, controversies. The pma, which I'm part of, the Measurements Council have been grateful for all the collaborations they've been doing in terms of educating folks in this space about the various stand on rules by networks. So the conversation around this topic has been significant. Even what was it past this summer prominent influencer really called out Honey for stealing my commissions. Those kind of 25 year plus pass controversies emerging again. So for those that are that are not following, this is a prominent affiliate. This is owned by PayPal. This affiliate Honey is a browser extension so you can use it as a, as a companion to your browser to find deals and find codes and attach those at checkout. And the consumers to their credit are often wanting that ease of use, wanting that, wanting that ability. But a lot of this comes into an issue when does the user know what they're getting into? Is that a fair credit towards the other affiliates and content pieces that got someone to make that transaction at a Nike, a Walmart, a software of purchase if you will. So just trying to frame what's happened here for folks to follow along. So Ben is providing the facts and the what happened around the case.
- [07:15]
A
The question of whether shopping plugins genuinely add value is a great one and your remarks ty go to the heart of that. Weirdly, the dispute of the past six weeks is not about that. I accept that networks and merchants think shopping plugins add some value, even if I disagree. No one asked me, least not recently, back when people used to ask me they could get my opinion, I'd give it for free. I'd charge them for it. My opinion is available. But if you decide to work with these plugins, they're supposed to follow the rules. And there are a set of rules which have the force of contract, contract between Honey and the networks and in turn the merchants promising to do certain things in certain ways and frankly not to steal other people's candy, not to jump in at the last second and claim commissions that were supposed to flow to others. Not only did they break those rules, as various software plugins have done over the years, but they intentionally concealed it in such a way that they were not likely to get caught. And they really weren't likely to get caught. They got away with it for years until finally some folks. Got myself lucky to be included.
- [08:14]
B
And for those that are not super into our industry, maybe you could give a sense of an example where in this situation, in a transaction with a customer and a network and multiple publishers, Honey is essentially in violation. What exactly does that look like from your perspective?
- [08:34]
A
So a stand down violation always begins with a user at a quote, unquote normal web publisher site. So the user is at New York Times Wirecutter, the product review site that a lot of people have used. You're at New York Times Wirecutter, you're reviewing Bluetooth, you're reading their reviews of Bluetooth headphones. You click one you like, it says it's going to send you to Newegg. It does send you to Newegg. You look at the product and up pops the Honey Shopping Extension. Honey Shopping Extension says something about Newegg. Maybe it says it as a coupon. Maybe it says you can earn Honey gold or points or what have you. You click on that, you complete your purchase. It is a real purchase. Real money changes hands. It's not like a fake transaction. It's a fake referral. Fake in the sense that the real referral was from Wirecutter, but Honey jumps in at the last second and claims commission on it. Now, if you just think about it for a second, Wirecutter has real expenses. They have employees. The employees have an office. They have a test lab. There's rent on the lab. There's got to be someone who comes and cleans the toilets. There's an inspector who checks that the smoke detectors work. They have real costs and health insurance. Everyone versus Honey. What exactly does Honey have to do? They get you to put this thing on the computer. And then they pop up at the last second. Honey has the much easier business as between Honey and the New York Times. Frankly, I care a little bit more about the New York Times. My client from 20 years ago, suing adware companies. But look, we all need journalism. We all need real product reviews by real product reviewers who actually get the products and test. One of the best things about affiliate marketing is the genuine content ecosystem that it funds or is supposed to fund. But to keep that working, we can't have the honeys of the world jumping in at the last minute and claiming those commissions.
- [10:17]
B
Exactly. So just to give the framing right, we've got, we'll call it a content site or a review site, like a wirecutter. There could even be multiple other partners involved in the referral, if you will, or assists, educating the buyer, possibly to educate them to buy something. And this hypothetical, perhaps not, but just to frame it for people. And so Honey's coming in and essentially in some scenarios, in this hypothetical scenario, they're actually coming in at the last minute and claiming credit. Now, there are specifics around that. There's a world where that is valid, right? There's a world where just playing it out with you, a user could say to themselves, wow, Honey provided me with a deal. We could make subjective judgments on whether that, whether that's valuable or not. There's a world where the rules could be applicable. And in a last click attribution scenario, the brand is set up, a last click attribution, right or wrong, then Honey would come in and get that credit for that sale. Is that correct?
- [11:21]
A
Rules could be written in a lot of ways. As both an economist and a lawyer, I guess I put on my lawyer hat. Here we don't have to speculate about what the rules could have said. Here we have actual rules that embody the actual understanding between networks, Honey and merchants. And the actual rules say you have to stand down in this scenario. Are there other ways the rules could have been written? Maybe we're smart enough to either know or think we know what a better set of rules might have been. Absolutely. I've got some ideas too, but that really wasn't the question in the last six weeks. The question is, did they follow these rules? Did they respect these rules or did they intentionally break them and hide the fact of that violation in order to make some extra money? And to that we have just a decisive answer, thanks to the. The quality of forensic work that was done.
- [12:07]
B
Let's. And let's talk about. Well, let's get to the Forensic work. But I also want to talk about the stand rules. So for those that are not familiar with them, when was that set up? How is that, how is that enforced? Where is that required? I love, I love to give people the insight into the stand down rule because I think there's some confusion around stand down and situations where stand down applies.
- [12:29]
A
I wasn't there. This is a little bit before I got engaged with affiliate marketing. I got involved professionally in 2004. The first citations I could find to stand down rules go back to 2002. Maybe someone else can find a citation even earlier. You know, that part of the Internet's history gets a little messy. The discussion forum that affiliate pros were using disappeared and all the archives, they are hard to get. The essence of these stand down rules, though, is a general consensus that in turn gets incorporated into documents that I believe have the force of contract documents you have to accept. If you are a software publisher in any of the big affiliate networks. Each network writes its own contract. They choose different words, but different words for the same idea. And the idea is that if a web publisher refers a user to a merchant, then a software publisher can't jump in at the last minute and claim to have referred that user to that merchant. You have to let the web publisher get first chance. Why? Fundamentally, because the rules require it. And why do the rules require it? Because it's some notion of fairness that it wouldn't be fair for the software publisher always to get the last chance to be the last click. Just too big an advantage and doesn't spend the merchant's money the way the merchant wants the money spent. The merchant wants to induce legitimate affiliates like Wirecutter, Web affiliates, normal affiliates, to promote, say Newegg. To my example, Newegg knows that if they had a bunch of software publishers that were claiming all the commission at the last minute, they wouldn't be very attractive to the wirecutters of the world. Wirecutter might prefer to promote, I don't know, Best Buy instead of. And so in order to compete with that other hypothetical program that doesn't allow software publishers, Newegg says, well, we're not going to allow the software publishers to take your commissions. Maybe they'll take some others, but they're not going to take yours. We will protect you. And by inducing web publishers to stay, they get more web publishers in equilibrium under competition. Of course, all the merchants say the same thing and then all the publishers choose the merchants they want. But still the rules are there to benefit a specific party. The Web publishers who really do want them enforced.
- [14:31]
B
And I think just to give context on networks because they are a pretty big player in this space, as part of the involvement with the PMA they've put out some good resourcing around the Partner Marketing association which is a helpful guide on affiliate and partner marketing and influencer marketing networks. Like from a link down to what trade double or CJ impact partner is. They've kind of highlighted where they stand in the no pun intended, stand down take and stance and policies that the networks have, which is obviously catalyzing a lot of the conversation because they're the one taking action to remove money from the network as of what, a week ago? A little over that. So do they have a stand on policy? Yes, there's a resource we can share out for folks but the PMA published this a while back and we've been involved in it. But most of the networks, most all of them have required stand down to support what you're saying. Meaning it requires that the downloadable software partner or has to stand down to a web refer to your point. Some of them have an optional stand down which isn't something we need to look at and see what that exactly entails and then PMA does a lot to break that down as well. So it's an interesting is it's just a network policy. Each one has a different policy, fortunately or unfortunately. So curious to get kind of your perspective on that piece as well.
- [15:58]
A
To me the policies are pretty similar. There could be ambiguities that are resolved slightly differently when you read the actual words of the various contracts, what's been most interesting is to look at the slightly divergent approaches to enforcement with respect to the specific violation recently uncovered merchants obviously, excuse me, networks varied in their speed, some acted faster. I think that's inevitable. It was a busy week last week with asw. What's the difference between an expulsion and a suspension? Nominally a suspension is temporary and an expulsion lasts forever. I don't know. Nothing's really forever in the software business or the marketing business. So when I hear they're kicked out. But for how long really? The last time that a major software plugin was kicked out of a major affiliate network, they were kicked out for a while and then they quietly came back in. So I don't really take expel to be that different than suspend. The action that I thought was actually most strikingly different than the others was Awin. Awin said that honey's going to have to pay the cost of cleaning this up. There could Be lots of costs. There probably are some lawyers involved at this point, maybe forensic accountants figuring out whose money was taken and how much and how to get it back to the people who lost it. There might be testers, people like me or AWIN might want to set up a lab. They might want multiple labs in different languages, in different countries, different IP addresses. You could do that with VPNs and proxies, or you could physically set up some different facilities in different places. Starts to get expensive. Who should pay for that? Should AWIN shareholders pay for that? I have to give that a no. This is a problem Honey created. Honey should be paying for it as part of cleanup. And you can imagine their protests, hey, we only made $10 million from this thing and you want us to pay a $15 million fine? And you're like, yup, that's life in the big city. You Forget to put $1 in the parking meter and you get a $20 fine, maybe $10 in the parking meter and a $200 fine. And that's kind of the way fines go. The fine is more than your ill gotten gains. The fine is supposed to deter and discourage. When you start thinking about it that way, there could be a lot of money for testers, there could be money for networks to convene thought leaders. Maybe they should bring you in, ty to advise them on what their policy should be and do it at Honey's expense and fly you first class because they should. It's a burden on you and take you away from your family and the rest of it. So costs will be high and AWIN to its credit says Honey will be paying those costs with an implied or else. If they don't, maybe they'll start to see a worse suspension or an expulsion or what have you.
- [18:26]
B
The specific evidence you uncovered, Ben, in terms of your expertise and your skill at catching fraud in the act, which is one of the few folks that are doing this, I think it's an important piece that we've talked about. A lot needs to be happening more frequently and think such a value that you bring to the community. In the case of Honey, in the timeline of like, how did that happen and like, what were the specific evidence that you found that showed they were replacing affiliate codes without permission and evading.
- [18:54]
A
The evidence was unusual for me, historically my main method has been, as you say, hands on testing. I get a copy of a piece of software and I test it in my lab. I make videos and packet logs so I can go back and slow motion, look line by line Packet by packet, what did they do? What was on screen? This was different in that I realized I could get the source code. A Chrome plugin is actually just a zip file with some JavaScript inside and you can read the JavaScript. Now the JavaScript is minified, which means all the function names and variable names are very hard to read. They've been replaced with gibberish. But still, if you work hard or you have the right AI to help you, you might be able to figure out quite a bit. I did some of it myself and got AI to help me with some and had to check the work. Obviously for a high stakes question like this, you can't defer to AI. Then with it running I could cross check with config files with telemetry. It was a much deeper dive than I've done with the other shopping plugins I've tested over the years. That reflected in part the nature of the violation, that a concealment violation is a more complicated violation than a stand down violation. Concealment requires thinking about the different conditions. If the user is 27 days old, what happens? And what if the user has 5,000 points? What if the user has 20,000 points? What are the thresholds anyway? To Honey's credit, a lot of this was really, it was in plain view, it was in their configuration files, it was in their telemetry, all using codes and abbreviations and shorthand. And there were a couple of places where they were definitely trying to hide it. They didn't want to put the affiliate network domain names adjacent to the cookie names. So rather than making a list that had two columns, if you're looking@cj.com, look for this cookie. They made two, two separate lists that they put in separate places. And you have to read the list together. If you're at row 2 in the domain name list, then check row 2 in the cookie name list. So they were hiding, but they weren't hiding as badly as some other programs that I've tested over the years. And I guess I'm grateful for that. This was hard enough, I didn't need it to be any harder. An interesting thing was that all the methods pointed in the very same direction. It's not like there was some contradictory evidence. There was some evidence that they were hiding and some that they weren't hiding. No, all the evidence said they were using these stand down tricks to avoid violating the rules for users who they reasonably thought were or could be testers.
- [21:05]
B
And I thought the notable thing, pretty wild, was that if I'm Correct. In my recall, they would only stand down for six hours in some cases. Is that correct?
- [21:17]
A
There were times when it was minutes. It was as little as six minutes at one point. Then they changed it to 60 minutes. Neither of those numbers appears anywhere in the contract. A contract says for a session and how long is the session? I think a session is until you close your web browser. I've had web browser sessions that continued for days. I don't restart my computer and I think that's still one session. Maybe someone would say if you go to sleep in the middle of the session, then it's actually two sessions. If you go and get lunch, then it's two sessions. Those might be reasonable arguments. Lawyers can fight about that someday, but.
- [21:51]
B
But not six minutes.
- [21:52]
A
Six minutes is clearly wrong.
- [21:54]
B
Six hours is dubious too.
- [21:56]
A
Yeah, six minutes is really just enough to trick a tester. A tester who wants to make one continuous five minute video is never going to catch it if your stand down is six minutes. I'm lucky in that my automation has always gone on longer time horizons than that. Vpt, the consultancy where I'm leading testing and forensics, again, they have methods to catch things that are much slower than that. But, but they were up to some tricks for sure.
- [22:19]
B
I mean, and I think, like, what does this mean? I mean, we can get into all the other things, but are there other factual pieces that you think are relevant that we should be talking about or looking at? I think there's a lot of implications around other similar partner types in terms of how much scrutiny they have. Does this. How does this change the Hate to even bring it up, but the legal battles that are happening, which I'm not necessarily directly involved in, fortunately, I'm trying to, trying to understand.
- [22:48]
A
Those are all great subjects. A couple of quick thoughts on that. The lawsuit gets a lot stronger with evidence that Honey not just was breaking the rules, but knew they were breaking the rules and hid the violations. If they knew they were breaking the rules, then it was definitely breaking the rules. Versus historically, Honey was arguing that they weren't breaking the rules, that the rules didn't mean what plaintiff said they meant, and the fact that networks couldn't have caught the violations because the violations were hidden. That also explains why the networks didn't catch the violations and why litigation is necessary to clean up this mess. As you say, there are other shopping plugins that are doing things that are kind of similar. No one has ever produced any evidence of other shopping plugins intentionally hiding from testers. That doesn't mean that I don't have some examples in my mind or maybe on my hard drive, but I don't know. I gave this one away for free. I may be done working for free for now. If anyone wants to know who else is cheating intentionally. I got bills to pay. I got property taxes. I don't have a mortgage anymore. Someone's got to pay for this kind of work, and we're all in business here. My angle is I get paid for the services that I deliver, and right now I need more clients to request this kind of work. Finally, the shoe waiting to drop is a fight brewing between Honey and its prior leadership, the people who started the startup and sold it to PayPal. Historically, the prior leadership of Honey has been pretty quick to defend Honey in social media when they've been accused of wrongdoing. Last week, Honey's statement was that this was was designed by a prior management team. So this is PayPal really throwing the startup founders under the bus. How are they going to react to that? The fact is, there's some evidence in support of what PayPal says. This does look like a system that was designed before the acquisition, but it was also updated and changed after the acquisition. I think there's egg on everyone's face in that regard. I'd like to see them fight it out. I'd like to see them point fingers at each other, give it their best shot, and see who reveals the most. Maybe they'll realize that they don't actually stand to benefit from blaming each other because it'll just make them both look guilty. But that is the path that PayPal was going down as of last week.
- [24:56]
B
Wow, that's some serious challenges and implications there and definitely concerning and not shocking, given the new information that was, that was, that was shared. Do you think that this is going to bring broader fraud control, network quality, regulatory scrutiny, or policy or standards in networks, law organizations like PMA that are helping educators? Do you feel like what do you think is going to come out of this from your legal and network quality view?
- [25:27]
A
I'd say the publisher class action is going to succeed as to Honey, they're going to get some money out of Honey and give it to legitimate publishers. I didn't necessarily think that the day the case was filed, but with these facts, they can thank me later. With these facts, they have a winning case. With the facts as they pled it. I don't know if they had a winning case, but now they do. And if they do a good job with the case, they should win and publishers should get money at Honey's expense. Honey's PayPal shareholders will be giving money to the publishers they took money from. I doubt government regulators are going to come in this is B2B context. Ultimately, even the influencer creator reviewer sites are acting in a business capacity. So I don't expect to see the FTC doing anything here in other countries. I think your audience skews American. Our customers skew American. Your customers. But in Europe, there's been greater skepticism of shopping plugins for a long time. Awin, of course, based in England, is particularly receptive to that. It would be a good thing if Americans got some of the European skepticism, if you ask me. Just in the sense that the Europeans spend their money a little bit more carefully, maybe when they're sending it to American tech companies, it feels particularly galling and so they want to avoid doing that to the extent they can.
- [26:43]
B
From our perspective on the agency side, for many years we've been extremely skeptical of partnering with browser extensions. We're very extremely limited in how and when we're doing it, and so we have to be thoughtful about it. I know there's a lot of brands that have sensitivity and scrutiny for obvious reasons. I think there's different flavors and different standards across them obviously from Honey to Rakuten Rewards to Capital One Shopping. They're not all necessarily the same, certainly, especially with the new findings that have you've unearthed and I think that's an important call up from my view. We take a very careful There's a lot of partner types available in our on our world in partner marketing and I think there's a lot of areas for us to work on, especially for the types of clients we work with. So it's not something that we've fortunately been pushing and leading a lot of to say you need to be integrating with a lot of downloadable software partners, which is great. It's not shocking to us to see this, but I think it's extremely helpful to educate the community, have good conversations that are challenging to make sure with networks with PMA with you, with network quality folks with brands with publishers to educate them. Because I think there is an education gap still and there's a perception gap. And so I'm grateful to have you jump on, do an emergency POD conversation and hopefully give people some intel that's helpful in this thing. And for those that want to like, reach out and learn more from your perspective, hire you, talk to you, understand you know more about this topic, what would you suggest and what other next steps do you suggest?
- [28:14]
A
I'M easily found online. Type my name into Google and up it pops. I'm also advising a company called vpt Visible Performance Technologies, where we test things like this with automation at scale. We have hundreds of examples of honey misconduct available to the lawyers and merchants who need it. And networks, I suppose. And we're protecting clients from this kind of problem day in and day out.
- [28:36]
B
Thanks, Ben. Thanks so much. Talk soon.
- [28:39]
A
Bye.