Loading summary
Kieran Rogers
Welcome to the Digital Marketing Podcast brought to you by targetinternet.com.
Daniel Rolls
Hello and welcome back to the Digital Marketing Podcast. My name is Kieran Rogers.
And I'm Daniel Rolls.
And today, Daniel, we've got an interesting episode. We're going to be talking about testing everything and I'm excited about this episode because we've actually got one of our listeners who got in touch and kind of inspired this whole episode actually by sharing some of his experience following our shout out a few podcasts ago for those of you listening. And yeah, I thought it was a really, really interesting insight that he's provided us. And we just thought it'd be really good to just explore this a little bit before we go into that interview.
So let's put it in context first of all. So we are going to talk about a B testing and multivariate testing, of which I will define both in just a moment, what they really are. But let me put it in context. So a B testing is obviously trying different content against each other to see which one works better, whether that is email subject lines or landing pages on your website or what we'll explore later on looking at Facebook ads. But the key thing here is to realize that we know nothing and essentially that you do need to test everything because you need to look at challenging our assumptions about things.
Can I, can I do a quote from Game of Thrones?
I thought this was coming when I said it.
You know nothing, Jon Snow.
I was expecting that. So the reality is that if I give an example, it puts it in context. You create two sign up forms on your website. This is a real world example, and they're trying to collect email data. And on one of them you've put the form and then you think to yourself, do you know what? If we put a security certificate on that form, people are going to trust filling in the form more, know that we're a secure website, and they're going to be more likely to fill in the form, which is actually a perfectly logical bit of best practice. It is best practice to do that. So you go, well, I'm going to do an A B test and prove that's the case. Now, in most cases, you probably wouldn't even bother doing the A B test. You would go, this is best practice. I am going to do it. So on this particular website in question, they went through, they put the certificate on there, but they did it as an A B test and they tested it out. And what they actually found is that the certificate version of the page versus the version of the page with no certificate on it, the one with the certificate got 12.5% less people filling it in.
Kirti Man Koholi
Wow.
Daniel Rolls
Because what happened is you go to the page, you see certificate, you start thinking about trust, and you think, well, maybe I shouldn't be putting my email address everywhere. Maybe I won't bother filling this form in. So it's completely counterintuitive. And that's why we need to do testing to prove everything. You will find that everyone in every organization is an expert in at least two things. One of them is copywriting. Because anyone who gives some copy, they will rewrite it and go, you should just change it to this. And everyone is an expert in web design because you show them a website and they go, you should do that bit there in blue and that should be a title, and so on. And it takes the subjectivity out of things. That's the key thing. So removing subjectivity is important. Let's just define a little bit AB testing versus multivariate testing. So A B testing may actually be ABCDE testing. Bear in mind, it doesn't mean you're just doing two versions, but you're taking an original version and then you're testing something against it. Normally using a system online to do that. And you might not test on your entire audience, but you might say 20% of your audience. Give one version 10%, one version 10%, and then see which one works best. And then you can decide to show that to the rest of your audience should you want to. Multivariate testing. Slightly different multivariate testing. For example, you could take five versions, the heading, two versions, the text, six versions, the image, mix them up, spew out different versions of that web page, and then decide which version of that web page is working better. Most of these tools will do the statistical analysis for you to work out what is statistically significant and what's not, rather than leaving that to you. But they help you kind of automate this process. Now, talking of tools, there has been a change in Google Analytics.
Yeah. So notice this a while ago for any of you that have sort of looked at kind of multivariate testing, A B testing, I should say there was a lovely little tool once upon a time called Google Content Experiments, which was all part of analytics, and enable you to set up some variation testing. It wasn't true multivariate testing though.
No, it was just AB testing. And it was under the behavior reports in Google Analytics and down the bottom it said content experiments. And you could do some A B testing. Now it's still there, I should say it's still actually there, but when you go in now, it'll actually say this is not the prefer way of doing a B test by Google anymore. This feature is being depreciated. And actually there is a new tool called Google Optimize.
And for those of you listening who maybe have some content experiments, data within Google Analytics, don't panic because it will still be there. If you've got existing experiments still running there, they're still there, but they're very much wanting to kind of move away from that older tool into one that actually does do the full nine yards and really enables you to run multivariate testing with a high degree of automation. And I think what's amazing is that this isn't just a Google Premium analytics feature. This is something that they're offering to everybody, which is great.
So if you want to find it, if you go into Google Analytics, you're logged in top right hand corner, you will normally find the little grid icon which shows which other tools you have available. You click on that and it will give you things like Tag Manager, Google Surveys, which is fantastic as well, which I'm going to talk about in just a second. Then you'll find in there the Optimize tool as well. Now, Google Surveys, you've got a good.
Story about this and I'm intrigued because I don't know.
Yeah, well, it's just a way of testing assumptions. So you can go to Google Surveys, you can build a survey like you would in SurveyMonkey or any of those tools as well, but you can deploy it. And SurveyMonkey, I should say, does this as well now, but you can choose an audience you want to target. So I would go through and say I want to target people in the UK that have the word marketing in their job, for example. So marketing managers, marketing directors, all those kind of people, digital marketers. What happened is on Target Internet, where we sell our E learning online training stuff. I realized when we were building the direct to individual offering. So if you're not aware, we have this kind of B2B offering we've done for a long time and we came along and said, look, we're going to let people sign up for this individually, stick a credit card in and they can get access to that. We said to ourselves, what's a good price point? So we looked at the competition and we worked out 25 to 50 pounds a month. Kind of fitted in with the competition. So we said, okay, let's set it at £25. A month. And then I had an argument with myself and said, well, actually if it's 25 to 50, why would we go the low end? We're better off selling at £50amonth and selling half as much and getting just as much money. So where should we really go? So all we did is we went out to this kind of tool to Google Surveys and said if you were paying for an online training resource and we kind of described the service a bit, how much would be a reasonable price to pay for it? And we put a lot of price brackets in and everyone came in and said 25 pounds is great. Now they would say that obviously, because cheaper is better than more expensive. But what it made me realize is it made it a bit of a no brainer. And we wanted to build numbers initially as well. So we said, okay, well we'll kind of start at that point. Now there's lots of argument about pricing where people will disagree with me now and they say you should never start at lower price. You should start higher because then you can half your price and all those kind of things.
I can see people listening to, shaking their fist that are on their phones.
But my reality is we said, okay, well what we're going to do, we're going to go in. And that's what people liked as a price point. So that's what we kind of started. But it made me sure we weren't plucking it out of thin air, which is exactly what we had done initially. And it goes back to the point they're all experts in something and normally it's our own business. Oh, we know exactly how it works and how it should be done. But we're not really thinking from a customer centric point of view or making assumptions about our customers. So I think Google Surveys is a nice way of again of testing everything and taking some of that subjectivity out of the mix a little bit as well.
I think at this point also it's worth mentioning our customers in many respects.
Kieran Rogers
Because actually.
Daniel Rolls
There'S a degree that you do know about your customers and certainly systems like Google Analytics have helped with this massively. But they do tend to look at everybody en masse. And in that with that kind of 10,000 foot view, you lose a lot of the detail. And we've recently been spending some time actually analyzing how people are actually using the web pages and interacting with individual web pages. And it's absolutely fascinating. You would never get these kind of insights from your analytics reports because they're looking at everything in aggregate but when you start going through maybe five or ten different users, actual journeys on the website, you start to see the power of multivariate testing. And actually your customers, when you look at analytics at that level, show you different variations and different. They're all using the same pages, but they're actually all using them in different ways. And analytics won't necessarily tell you that you sort of have to. And I know I bang on about this all the time and I'm really sorry, but you have to walk in your customer's shoes and there are tools out there that enable you to do that really, really effectively.
So we're using hotjar on the website which allows us to record user sessions and then play them back. And it is amazing when you watch it back actually seeing where people are misunderstanding things, they're really not making the assumptions that you thought they would and so on as well. And what's really interesting about this is that analytics makes us forget a key principle of usability. And one of the things that I always preach about in usability is if you do even informal usability testing, a lot of the stats show that if you get five people doing the same thing, so you ask five people to carry out the same task, you will probably identify 85% of the problems of your website. So five people doing the same thing, identifying 85% of the problems of your website. And that completely holds true. But then what happens is we look at analytics in aggregate and we miss all these individual problems. So actually when I've showed people hotjar, it just records what people are doing on your website. So this must be quite time consuming. It is, but it's hugely valuable because you sit there and go, okay, they did that. That person a minute ago did that as well. Right? We need to. So then you go on and say I think we should change this, then you can a b test it and then you're in a really nice path of improving things for your actual customers rather than groups of customers.
But it's sort of another way too, isn't it? Rather than actually you have customers doing multiple things on your website all of the time. So it's kind of, I guess that's why I mention it. It is kind of multivariate testing of a fashion. The other point that I wanted to make on that is that you may more eagle eyed amongst you may have noticed that when you visit the Target Internet website, which is wgetinternet.com There's a little orange feedback tab on all of the blog posts and public facing pages. And I'd encourage you all to use that, because do you know what, if you want to get a message to us about something that doesn't quite work right, or something that you found frustrating, fill it out and I will be all over it. I will be on it like a bonnet because I'm monitoring those all the time. And actually, it's great to get your feedback because actually we want to make the website better and we do want to make sure you're giving good experience. So please, please make use of that and give us some feedback.
Yeah, and I think it's important, actually, just from a business point of view, if you're going to have a feature like that, you get back to people fairly quickly as well, because otherwise people have this real feeling, there's no point filling those in because no one's looking at anyway. No one's going to do anything about it. And actually all you do, even if you've got problems, you build a personal relationship all of a sudden, because someone said there's a problem, you're going to go and fix it, they like you a bit more. And it actually is positive part of the user experience as well.
So. So we have an interview. Yeah. So what is interesting? So this is a listener called Kurtiman. Kurtiman got in touch with us from India. I got very excited. I knew we had listeners in India, but we'd never spoken to any before. And basically when we put a shout out, say if anybody's got anything fun they've been working on that they'd like to share, Kurtman got straight back in touch. And he works for one of India's largest gadget research websites, and they've run some really interesting stuff, kind of reviewing different gadgets and stuff. And what he done is he's applied multivariate testing effectively to his Facebook advertising. And I think the insights that he's got are really, really interesting. It's a really interesting case study that I think everybody should listen into. So, without further ado, here's the interview.
Kieran Rogers
Hello and welcome back to the Digital Marketing Podcast. My name is Kieran Rogers and I'm excited today because we've got something a bit different for you. A few podcasts ago, we put out a shout asking people to get in touch if they had anything thought other listeners might like to hear about. And we have one such person on the line today, all the way from India via Skype, and his name is Kurtiman. Hello, Kurtiman.
Kirti Man Koholi
Hi, Kiran, how are you?
Kieran Rogers
I'm all right. And Kurtiman, you're a. You're a podcast listener. How did you come. Yes, how did you come to find out about the podcast?
Kirti Man Koholi
So I've been a podcast last few months actually. So I, when I travel for work, I drive around four hours a day and I used to listen to a lot of music earlier but now started utilizing my time to listen to podcasts and I listened to a lot of podcasts earlier, but I found your podcast particularly very relevant to the work I do. So I've been a fan since then and I've been regularly listening to podcasts since then.
Kieran Rogers
Fantastic. Well, thank you so much for getting in touch. I suppose really we should start by encouraging you to introduce yourself. Really. So tell me your full name and what you do.
Kirti Man Koholi
So my name is Kirti Man Koholi and I'm a digital marketing manager with almost five years of expertise in social media marketing, performance marketing growth, hacking and E commerce. So currently I'm working with 91Mobiles.
Daniel Rolls
Okay.
Kieran Rogers
And tell me more about 91Mobiles. What sort of work do they do? What sort of content do they work with?
Kirti Man Koholi
So 91Mobiles is India's largest gadget research website. And along with 91Mobiles, we also have a property by the name of Killer Features. So collectively, we do a lot of technology content focusing on a lot of latest smartphones, all the gadgets, and everything that goes around in the technology space all around the world with killer features. What we're trying to do is we're trying to promote a different take on technology which is not. Which does not include all the nitty gritties. It's more of the lifestyle angle of technology, focusing on the coolest gadgets, the latest trends in automobiles, cutting edge innovations and new media trends.
Kieran Rogers
Great. And the thing that you and I got chatting about was some of the challenges you were facing with your social media advertising and trying to find ways of actually getting, getting that to work. Can you share with us what the challenge was there?
Kirti Man Koholi
So we started killer features around 10 months back. And so we started early on, we started spending a little money on a few social media channels to get readers to our blog. So with a small budget, we used to get a great cpc. We used to get users, relevant users who used to engage a lot at a lower price. But after a while, when we wanted to scale this, the CPC went up exponentially. So trying to get a large number of people coming to the website, a large traffic at a lower CPC wasn't possible after a while. So we couldn't control the CPC on.
Kieran Rogers
A Large scale, I think that's a problem a lot of people have. You know, when you first start out, your cost per click at a smaller scale can be quite reasonable. But with a lot of these platforms, as you try and push a message out to a much, much wider audience. So because it's less targeted, the costs can escalate quite quickly. And suddenly it doesn't seem like the one size fits all, fix it solution that everybody keeps telling you that all these different platforms are. So how did you respond to this? What sort of things did you try and what were the solutions you found that worked?
Kirti Man Koholi
So to counter this, we tried a lot of techniques, we tried experimenting with a lot of audiences, we tried doing retargeting, tried interest based targeting a lot of audiences. And along with that, we also did a lot of experiments with the packaging of the content, trying a lot of different images, a lot of different headliners, short form, long form, trying to see what works. We also approached a few guys from all around the world who had a few social media tools and a few experts, but nothing gave us any sustainable results.
Kieran Rogers
So what was the sort of breakthrough moment within the campaigns when you started to get to get results?
Kirti Man Koholi
So I had a bit of background from the product side as well, and we did a lot of AB testing, but that was only restricted to a lot of elements on the website. But we didn't, we never tried AB testing for social media or any of the content that we did. So I came across a tool by the name of adespresso. And adespresso basically is a Facebook ad management tool which helps you run a lot of AB tests on all the posts that you do on your social media, Facebook particularly. And with that you can create multiple variations of the same ad and you can just put a few headliners, put a few images across and the tool does the rest.
Kieran Rogers
Okay, what was the overall effect on your results of doing this? It's interesting that you sort of started to see some success using a B testing on your ads rather than necessarily on your landing pages.
Kirti Man Koholi
So what we did is rather than spending more money on the same post, we divided the same content into various variations. So each content had, let's say, five image posts and five headlines. Five images and five headlines, and you multiply that, it's 25 variations altogether. And after running all those variations with small budgets in the next 24 hours, reaching 500,000 people per variation, we could know that to what variation is the audience responding and what is yielding better engagement and best results at a lower cpc and as soon as we got that in the next 24 hours, we started spending more money on the variations that were performing better.
Kieran Rogers
Were there big variations in the results?
Kirti Man Koholi
So over a period of time we started having a few learnings, we carved out a few trends for us. So first big trend that we saw was that shorter headlines with good numbers, with hard hitting numbers and stats used to work a lot and good looking HD images work a lot. So rather than having a basic image that doesn't tell a lot of story, having a good HD image of the product that you're trying to promote or the content, a relevant image for that content works a lot, really.
Kieran Rogers
And actually having it in high definition made a difference.
Kirti Man Koholi
Yes, it did. And also what we did is we created a small library of images which we could repurpose on a consistent basis to derive better results.
Kieran Rogers
Great. And so in terms of the results that you got, can we break your learnings down into a bit of a formula? Is there sort of a step by step approach that others could try out? I'm conscious that obviously that worked for your audience who are sort of primarily a tech based audience, but I think it's great to try and break these things down into a little bit of a formula based on what you kind of learned.
Kirti Man Koholi
So in essence, people come on Facebook to break the monotony, to read something which is interesting, which invites some sort of engagement. So having smaller headlines and looking images worked a lot for us. And what I would advise is people from other industries, they could take this tool, it's not a very expensive tool to have at Espresso. Create a few variations of the headlines, discuss it internally within the team that what different variations can you have for the same content or maybe of the same product? If you're selling for the headline and for the images, create a few variations there and it's better to get all the relevant learnings yourself. And over a period of time, maybe a month or two months time, you can get all the learnings you want.
Kieran Rogers
So tell me a bit more about Ad Espresso. How much did that contribute to what you were doing?
Kirti Man Koholi
So when we started with Ad Espresso, we started with a very basic plan, doing very few campaigns, but over time the results were so phenomenal that we've shifted all though. We don't have a lot of paid traffic, lot of traffic is organic, but we focused most of the paid traffic coming from Adespresso as well. We have taken our traffic up to 5x with just a marginal increase in.
Kieran Rogers
CPC really five times.
Kirti Man Koholi
Five times, yes. And not taking a hit on the CPC much.
Kieran Rogers
Right. Okay. And I guess that's one of the real powers of paid for platforms. Organic is great, but you do tend to reach people that you reach by its very nature. But I guess with the paid for platforms, you can start experimenting with targeting different audiences and reaching new audiences. So it's a great way of just introducing yourself and what you offer to huge new segments of the marketplace.
Kirti Man Koholi
And one great thing that paid gives you is, for example, if you're trying to promote a new product or maybe some new segment of content that you want to promote and your old audience might not be the most relevant audience, so you can take that content and promote it to the new users or that kind of users that you want to promote your content to and extend the reach.
Kieran Rogers
Fantastic. So what do you think you're going to be tackling next, now that you've sort of got some of this success under your belt?
Kirti Man Koholi
So the next battle that we have is to scale the organic community that we have on all the social media platforms and to derive more engagement.
Kieran Rogers
Well, Kurtamin, thanks so much for giving us insight into what you've been up to. If any of you are listening out there and you've had campaigns that have been successful or giving surprising results, please do get in touch. We always love to hear about what you're all up to and how you're achieving what you're doing.
Kirti Man Koholi
Thank you so much.
Kieran Rogers
Thanks for listening to another episode of the Digital Marketing Podcast brought to you by Target Internet. If you'd like to get more information on the show, get hold of back issues of this podcast, or get details on any of the links we mention, please visit our website at www.targetinternet.com. if you've enjoyed the show, we would love to read your feedback. Please rate us in itunes or even better, write us a review. Or if you have any questions, please get in touch. We'd love to.
This episode dives into the crucial practice of "testing everything" in digital marketing—challenging assumptions through rigorous A/B and multivariate testing to drive success. Hosts Daniel and Ciaran share their own experiences and insights on testing methodologies, highlight the importance of moving beyond received wisdom, and feature an in-depth case study from listener Kirti Man Koholi in India about optimizing Facebook ad campaigns for better results.
Challenging Assumptions:
The hosts emphasize that best practices aren’t always best for every context and highlight how real users often behave unexpectedly.
“The key thing here is to realize that we know nothing and essentially that you do need to test everything because you need to look at challenging our assumptions about things.”
– Daniel Rowles [00:50]
A/B Testing Example:
Daniel explains a real scenario: Adding a security certificate badge (a common best practice) actually decreased form conversions by 12.5%. The badge, intended to increase trust, made users more suspicious.
“You see [the] certificate, you start thinking about trust, and you think, well, maybe I shouldn't be putting my email address everywhere.”
– Daniel Rowles [02:33]
Removing Subjectivity:
Both hosts call out how everyone in an organization feels expert about copy and design, but only testing provides real evidence for decisions.
A/B Testing (Split Testing):
Comparing two or more variations of a single element (e.g., email subject lines, landing pages).
“A/B testing may actually be ABCDE testing… but you’re taking an original version and then you’re testing something against it... Normally using a system online to do that.”
– Daniel Rowles [03:00]
Multivariate Testing:
Testing multiple elements and combinations simultaneously (e.g., mixing different headings, images, and text). Useful for more complex optimization.
“For example, you could take five versions, the heading, two versions, the text, six versions the image, mix them up, spew out different versions of that web page, and then decide which version... is working better.”
– Daniel Rowles [03:47]
Automation Tools:
Multivariate testing platforms help determine statistical significance automatically.
Google Content Experiments vs. Google Optimize:
The landscape has shifted from Google Content Experiments (more basic A/B capability, being deprecated) to Google Optimize (more robust, allows full multivariate testing).
“There’s a new tool called Google Optimize... It’s not just a Google Premium analytics feature. This is something that they’re offering to everybody, which is great.”
– Daniel Rowles [04:53]
Google Surveys for Pricing & Market Research:
Daniel shares using Google Surveys to set pricing for their online training—a simple, scalable way to test market responsiveness.
“What it made me realize is it made it a bit of a no brainer. And we wanted to build numbers initially as well. So we said, okay, well we’ll kind of start at that point.”
– Daniel Rowles [07:16]
From Analytics Aggregates to Customer Journeys:
While Google Analytics is useful, Ciaran and Daniel make a case for deep-diving into real user journeys using tools like Hotjar.
“You would never get these kind of insights from your analytics reports because they’re looking at everything in aggregate but when you start going through maybe five or ten different users, actual journeys on the website, you start to see the power of multivariate testing.”
– Daniel Rowles [08:27]
Usability Principle:
Even informal testing with just five users can reveal 85% of usability problems.
“Five people doing the same thing, identifying 85% of the problems of your website. And that completely holds true.”
– Daniel Rowles [09:34]
Real-Time Feedback:
Encouraging direct user feedback via the website’s feedback tool for continuous improvement.
“With a small budget, we used to get a great CPC... But after a while, when we wanted to scale this, the CPC went up exponentially.”
— Kirti Man Koholi [14:55]
“...create multiple variations of the same ad and you can just put a few headliners, put a few images across and the tool does the rest.”
— Kirti Man Koholi [17:07]
“Shorter headlines with good numbers, with hard hitting numbers and stats used to work a lot and good looking HD images work a lot.”
— Kirti Man Koholi [18:35]
“We have taken our traffic up to 5x with just a marginal increase in CPC.”
— Kirti Man Koholi [20:54]
“It’s better to get all the relevant learnings yourself. And over a period of time... you can get all the learnings you want.”
— Kirti Man Koholi [19:57]
On Assumptions:
“You know nothing, Jon Snow.”
– Daniel Rowles, referencing Game of Thrones [01:32]
On Removing Subjectivity:
“Everyone in every organization is an expert in at least two things: copywriting and web design.”
– Daniel Rowles [02:50]
On Listening to Real Users:
“You have to walk in your customer’s shoes and there are tools out there that enable you to do that really, really effectively.”
– Daniel Rowles [09:02]
On Adopting Testing as Routine:
“Always test everything... get all the relevant learnings yourself.”
– Kirti Man Koholi [19:57]
This episode offers a practical blend of strategic advice, hands-on tools, and a real-world case study to make the case for a testing-driven culture in digital marketing. Whether you’re optimizing a landing page or scaling paid social campaigns, “test everything” remains timeless advice.