← Scrum Master Toolbox Podcast: Agile storytelling from the trenches
Scrum Master Toolbox Podcast: Agile storytelling from the trenches
BONUS: Gojko Adzic on Optimizing Products for Long-Tail Users (Agile Online Summit 2024 Replay)
00:40:11
BONUS: Gojko Adzic on Optimizing Products for Long-Tail Users (Agile Online Summit 2024 Replay) In this BONUS episode, we revisit Gojko Adzic’s insightful interview at the . Gojko, an award-winning author and software expert, unpacks the...
Loading summary
Transcript45 lines
- [00:04]
Vasco Duarte
Hey, how are you doing? I'm Vasco Duarte, your host on the Scrum Master Toolbox podcast. And I've got some exciting news. So right now, as I record this, I'm holding in my hand the signed contract for our very first Global Agile Summit. We're all in and I couldn't wait to share this news with you. So mark your calendars. May 18th, 20th of 2025 in Tallinn, Estonia. We're gonna have a transformative experience. We're putting together an event that is all about real life Agile. It's not theory or buzzwords. It's practitioners sharing what's working, what's making an impact, and how they've overcome challenges that you too will have to face, or maybe even facing. Right now, we're bringing together the best stories in Agile. From product leaders to engineering wizards to business visionaries, these will be stories that will inspire you to action. This isn't just another conference. It's a chance to connect with the people that are shaping the future of Agile. And here's the best part. Right now, we're in our super early bird phase. And that means you can grab tickets at just 25% of the final price. Look, that's not just half off, it's half off of the half off. It's an incredible deal for our dedicated community members, just like you listening to this right now. So at the summit, day one will be all about hands on workshops. And days two and three, we'll dive into leadership, product strategy, coding, testing, and everything that makes Agile thrive in organizations. Right now remember, these are all first person, real life stories. Now whether you're a leader, a developer, or part of a consulting company, this event is built to take your Agile game to the next level. So don't wait. Go to globalagilesummit.com and grab your ticket. Today, let's all make 2025 the year agile truly transforms your teams, your business and our industry. I'll see you all in Tallinn. And Remember, go to globalagilesummit.com and get your super early bird ticket right now. It only be available until the agenda is announced, so don't wait. Grab it right now. Right now that that's out of the way, onto the episode. Hey there, agile friends. In this bonus episode, we have a very special guest, someone I've followed for a long time. That's Goiko Azic. This interview was for the Agile Online Summit 2024. If you want to know more about the Agile Online Summit, check out Agile Online. And of course you can subscribe to our community membership to get all the videos from the Agile online summit from 2024, including this one with Goiko Adic. But now let's go for the interview with Goiko about his latest book. If you're a product owner, you will certainly want to listen to this interview. His latest book is called Lizard Optimization. Oh and check out the Link for the 50% discount you can get on the book. It's all in the show notes. But now onto the episode. Welcome Agile friends. Welcome to this special interview with someone I've admired for a long time and followed his work, Gojko Atsic. Hey Goiko, welcome to the summit.
- [03:44]
Goiko Atsic
Hey Vasco, thank you very much for inviting me.
- [03:47]
Vasco Duarte
So Goiko is here to talk about something he discovered while developing his own product. He eventually named it Lizard Optimization and it's now a book. So for those of you more interested in the book, you can also find a 50% discount link in the video description below, so make sure you check that out. Goyko is an award winning software expert, Author and also 2019 AWS Serverless Hero. Why did they choose that name for the award? That's quite an amazing name.
- [04:22]
Goiko Atsic
Well, I was one of kind of the early people adopting Serverless and Robert Chatley and I actually wrote a relatively influential scientific paper on it. It's not something you'd expect from me to do and I kind of wrote a book on what we've done there as well. One of my products was a very, very, very early adopter of Serverless and then that led to some open source tools for that while the whole platform was still growing. And I think that's what they recognized as a community contribution, I think.
- [04:58]
Vasco Duarte
Absolutely, absolutely. So Goiko is also known for his influential work on agile testing. I bet many of you know him from that side as well. Serverless computing, as I just said. And one of my favorite product tools, impact mapping, something that I still talk about on a regular basis these days with the product teams that I work with. Now, before we dive into the conversation with Goyko, let me tell you that there's a lot more sessions on product topics in the summit this year. We have a whole track called Value Centric Product Development with Michal Barkova dedicated to product ideas. So check it out@agileonlinesummit.com but now let's focus on our conversation. So for those of us who are watching this session but haven't looked at many of your books, especially not the latest ones, we might even be surprised by the fact that you use illustrations quite a lot in the book. So one of the things that I'm curious about, Koiko, is why did you start using this specific style that it's more like a cross between nonfiction, children's book. Why did you choose that style for your latest books?
- [06:18]
Goiko Atsic
Well, I think books should be nice and easy to read, and a boring book, it doesn't matter how smart it is, is still a boring book. And part of kind of building a good product, whether it's software or book or something like that, is to focus on the usability of it and the value people can get from it. And I think visual illustrations can complement learning quite effectively and can cement in and reinforce learning. And I kind of don't take myself too seriously, so my books reflect that. I think having playful illustrations is really good. I used to work for a book publisher when I was younger, and I was actually editor in chief for a reasonably influential magazine about computing in Serbia when I was in my 20s. And we had this whole kind of thing about providing illustrations and visual reinforcement there. And then when I wrote Spec By Example, I worked with the publisher and then I was trying to get them to do some illustrations. As a publisher, they should kind of, you know, pay for illustrators and things like that. And there's some illustrations there, but I wasn't really happy about that and I wasn't under control of that. And then when I did Impact Mapping, that was the book after Spec by Example. I really wanted it to be high on usability, to get people to consume the content easier. And I started drawing lots of illustrations myself. And I sent it to a friend of mine who runs a design agency, and I said, look, I can still draw. I used to be able to draw. I can still draw. And this friend of mine replied with a single sentence, said, well, maybe you should pay a professional to do this. And I said, okay, fair enough. Find me somebody. So because this friend of mine works with lots of illustrators, he put me in touch with a bunch of people. And I really like the style of the illustrations that Nicola Korach was doing. And since Impact Mapping pretty much has illustrated all my books, it was wonderful cooperation, even on this one. And I think we are, because we've been working together for so long now. He knows me, I know him, and it works really well. Yes.
- [08:48]
Vasco Duarte
So when you think about this book, and of course viewers can also check out the other books, I'll put the link in the video description for people to find them. When you think about these books, as you said, they have a lot of usability in it. The format is slightly different than what you would expect from a nonfiction book. There's a lot of illustrations and that kind of is for those who haven't read the book. That is kind of also the perspective you take with the whole lizard optimization. It's kind of this oblique, not obvious perspective on that particular application of the method or on that particular method. So what got you started with this whole idea of what you call in the book lizard optimization.
- [09:34]
Goiko Atsic
So I build my own products now, and with that pretty much I do everything from pre sales to sales, product management, development, testing, all the way to kind of making coffee. And I'm pretty good barista now. The idea kind of came out of figuring out how do I improve the usability of the product, in particular for people that were struggling to use it. And it's a kind of mix between all the exception tracking, error monitoring, and that kind of observability that we're now doing in the technical spaces and industry and kind of figuring out where the value is from, impact mapping from all these kind of other product methods where effectively I was trying to figure out where are people really struggling with my product and how do I polish those rough edges and looking at what's unexpected. A lot of what happens in product management today, which is really, really good, is kind of people are going for measurability and metrics and things like that. But most analytics tools for product people are looking at expected things. They are showing page views, clicks, funnels, that we're looking at what's expected. But if you look at what we do with DevOps and observability, a lot of it is looking at unexpected stuff. If you have 1000 virtual machines running in the cloud and one of them starts to power cycle and crash, we want to know that. We don't know why it's doing it, but we want to know that so we can address it and fix it and things like that. We've developed tools that can help us look at exceptions, look at all these things, but those are not really used for product management. Those are used for technical stuff. And I started looking at that a bit more holistically and tracking what people are doing weird and unexpected with the product. And the idea with lizard optimization is to use this kind of analytical tooling to figure out when people are doing something that you can't understand why a rational human would be doing it.
- [11:59]
Vasco Duarte
I know many of those and you.
- [12:02]
Goiko Atsic
Know now and things like that. Well, you can imagine you get like a weird Bug report. And you know, I've been in many teams where people discard this as saying, well, the user is too stupid, or, you know, they're not technical or something like that. But it's not, it's actually a mismatch between what the software does and what the user expects or the way the user expects that. And then the interesting thing to figure out is, well, why or what. So, for example, one of the products I'm working on helps people create videos from PowerPoint files. And you type the voiceover script into the speaker notes, you upload it, the thing does text to speech and gives you a video. I discovered a bunch of people that were building blank videos and they were paying me for this, which is like, why would a rational human be paying us to create blank videos? And my first thought was, it's a bug. We have a bug in the pipeline. People have automated this. They don't see, they're not complaining. They're probably publishing blank videos somewhere without knowing that. And that's a problem. So I started investigating that and PowerPoint. I mean, there's a million tools that export to PowerPoint. So my first thinking was that they're using some weird tool that is not really saving it that way, or the library we're using to import PowerPoint is incompatible with that. And I couldn't figure it out. And then I kind of got in touch with all of those people and I said, well, when you open this PowerPoint on your machine, like, what does it show? And he said, well, it's a blank slide. And I'm like, okay, so like, why are you using some third party tool, open source, whatever? No, no, PowerPoint. And I'm like, wait a minute, man. I mean, so this is intentional. Yeah, yeah. So why? I mean, and it turns out that I've iterated on the text to speech usability so much that people were actually creating blank videos just to use the text to speech thing to extract an audio track. They didn't need the video, they only needed an audio track. And then, so that's, you know, the user is not too stupid. They're not creating blank videos because they're stupid. They're creating blank videos because they're getting value from the product in a way I didn't expect. And that's kind of the idea of a lizard, where somebody has kind of their own lizard logic for doing that. And then figuring that out got me to create a screen where you just upload the Word document and you get a text to speech. It's Much cheaper for us operationally to make an audio instead of building a video. It's much faster for the users. It is generally kind of just a win win. Yeah. And this is what kind of cat Holmes in her book Mismatch talks about, where you think of something like that that's a mismatch between a user and the software. You kind of really solve for one, but you expand too many. And you know, we didn't create the case how to make blank videos faster. We made a case how to make text to speech easier for people that need only audio tracks. And a month later, people were building as many audios as they were building videos. So the video usage didn't go down. The audio usage got up six months later. 99% of the stuff that was done on the system was audios. So we got 100 times growth by doing this, by figuring out a lizard and kind of helping the lizard succeed.
- [15:22]
Vasco Duarte
So we want to explore that a little bit more because you talked about this lizard logic and of course the book is called Lizard Optimization. But why lizard? Where did that metaphor come from?
- [15:33]
Goiko Atsic
I read a very nice blog post by Scott Alexander, who writes the Starslate Codex blog. And in the post he talked about some research they were doing that mixed demographics and some kind of psychological research. So they would ask you a bunch of questions about your demographics, like male, female, living in Finland, wherever, and then they would ask some psychological questions and then try to kind of correlate the patterns. And every research they've done, there was a bunch of answers they had to throw away because they were just ridiculously invalid. And he gave an example of somebody who wrote American as gender, which, I mean, okay, gender politics are a bit toxic now. Everybody can select their own gender, but American is a really weird gender, isn't it? And then there was kind of somebody who selected male as the nationality. So that's kind of the exact opposite of, you know, and so you have a male passport and a female passport and whatnot, you know, and, and so, and, and he talks about how on the face of it, you think, well, these people are stupid. They're kind of, you know. But actually there's a million reasons why that can happen. They might be distracted when they were filling in the form. They might be, you know, in, in an environment where this bright sunlight, like you're outside, the sun is shining, you don't really see on the screen what you're doing. Or you might, like, the screen might be small. You think you're answering one question, you're answering Another, and there's a ton of reasons why that can happen. And when you add up all these people, they're not an insignificant group. The long tail goes very, very long way. And when you add it up, it's a significant group of people. And kind of he talks about how basically he invented this lizardman constant, saying that it's like 4% is people doing things you cannot explain, but you either disregard them if you're doing research, or in case of product management, you think about, well, there's 4% of people here, we could be satisfying better. And it's something really, really interesting to explore because if you look at churn is one of the biggest problems people have in software. And a difference between a churn of 5% and 6% is probably, you know, millions of dollars for a product where if you have 4% of lizards doing something weird, helping some of them is going to increase retention quite significantly.
- [18:11]
Vasco Duarte
And the interesting part of that is, of course, because the 4% are just the tip of the iceberg. Because once you tap into the need, you might find, actually it isn't.
- [18:19]
Goiko Atsic
You might find a complete, you know, lizards can bring their friends if you're lucky.
- [18:24]
Vasco Duarte
I like the metaphor.
- [18:26]
Goiko Atsic
I mean, so you know, one, one very practical example of how this kind of maps all together with observability and monitoring and things like that. So usually a good software, if you have a user doing something that is unsupported but predictable, the software presents a decent error message. So it's not like, oh, unexpected error 057333 cancel or okay, that's ridiculous. Something that helps the user resolve the situation. So for example, going back to the screen that allows people to upload the word document, you select a file that doesn't contain text. People occasionally do lizards, they do it on a phone where they click on the wrong file or something like that. So I have a decent percentage of people every day selecting JPEG files, photos, this kind of. So the error says, hey, you selected a photo, you probably did not want to do that, select a different file. But in addition to that, what you would expect a regular software to do, I also get the message that somebody select the JPEG file. And this goes into this kind of monitoring thing. We look at what's the unexpected thing that people are doing. Some percentage of people every day upload MP3 files to a text to speech system. It doesn't make any sense. It doesn't make it, it doesn't have any logic. It's already an audio file. Why would you do that. Like my best guess is, you know, you have these like indie singer songwriters that want to get their music out. I don't, I don't know. And, or I have some percentage of people that are uploading APK files, Android applications to a text to speech system. How? Why? I mean, what do you expect from it? It's really, really weird. Lizards. You cannot explain it, but there was a percentage of people every day uploading subtitle files.
- [20:27]
Vasco Duarte
Oh yeah.
- [20:32]
Goiko Atsic
I can see the logic here. I mean it's a text file. I started actually enabling that. It's just a text file, why not let it do. And then I started talking to some people who were actually doing it. Get in touch with the lizard, see if he's doing what they need to do, verify it. And one guy said well yeah, it's, it's, it's kind of reading the timestamps. And I said well it's reading the timestamps because the subtitle file has the timestamps. You've uploaded a file with timestamps. It's reading the content. That's what the system does. It reads the content. They said, but I don't want it to read the timestamps. I wanted to read only the kind of the text. Yeah, a subtitle, narration. And I said okay, that, you know, that makes sense. I didn't expect that. Let's remove this timestamps. And then I, I got in touch with him again. I said, but now it's talking too slow, civil. Okay, there's, there's something here I don't understand. Like what's the job you, you want to do with this? What? What? Oh, he was creating an alternate audio track for his video. He had a video in English. He wanted the video in French. He doesn't speak French. He wanted to use a text to speech tool to translate English to French and get, ah, now we're getting somewh what's going on. And I mean it's not a trivial easy thing to do, but why not? I mean it, it sounded like fun and it, it was this like 1% to 1% case. But hey, let's help people do this. So I've actually kind of, you know, it does, does take a bit of, of tech smartness to, to do that. A couple of days later I had a version of the software. If you upload the subtitle file, it actually synchronizes the audio with the timestamps. That was by far the most profitable thing I've ever done in my life.
- [22:28]
Vasco Duarte
Oh, wow.
- [22:29]
Goiko Atsic
Because a couple of weeks later, one of kind of very, very large enterprise software company started using it. They discovered it and they had hundreds of thousands of videos that were instruction videos for their software. They had them in English, they had the subtitles, they wanted to translate to dozens of languages. And it's not a massive use case. If you look at the number of users that are using it, it's tiny. But if you look at the profitability of it, it's insane. And the funny thing is 99% of the functionality was already there. As you said, it's the tip of the iceberg. It's just kind of getting it.
- [23:07]
Vasco Duarte
That's why they were using it, that's why you had the lizards there.
- [23:11]
Goiko Atsic
I would have never discovered that through user research because it's such a tiny percentage. It's an absolute, it's a freak.
- [23:20]
Vasco Duarte
This is a very good point because user research, no matter how good it is, it's always directed. There's an assumption of what we are looking for. But the example you just shared is undirected. It's unprompted insights.
- [23:35]
Goiko Atsic
And that's what I said. It's complementing this kind of feed forward that's user researcher things. That stuff we're looking at what's expected with what kind of weird shit is happening in production that we don't even know. And that's an interest. So I'll give you another example. The error monitoring and exception monitoring picked up that a bunch of users were trying to create videos with unsupported parameters. And not just unsupported parameters, but repeatedly same unsupported parameters. Like instead of the voice parameter they would be sending voice, but in French, like voix and things like that. It was weird because I couldn't figure out how they're doing it. So these tools now called session recorders, where you can install a snippet in the website code and then it captures the DOM and then it presents it back to you as a video. I turned it on after users have this error because then you can figure out that's the user session and focused on that. I was like completely shocked to see my website translated in a different language. It's like we don't have that. I mean, is this somebody copying it? Is this what's going on? And it was actually served on a totally different domain. And I figured that kind of the way that's happening is if you load the website and you have some kind of Chrome plugin, whatever Chrome offers you to translate it, and it goes to. There's two ways of doing it. One way of doing it is where the translation happens in the browser. The other way of doing it is where Google Translate does some kind of weird proxying where they create a website, some kind of proxy between the user and you, and then they translate stuff as it goes on. Now what it did is it started translating configuration JSON files, not just the website, but all the kind of cold everything. Yeah. And that caused all the weird parameters and that's like an insane case. But then we figured, okay, we have people who are using this in a translated context. I didn't expect this, but it's not necessarily something we should be fighting against. I mean, it's, you know, is it a bug? Well, it's not a bug. Is it a mismatch?
- [26:19]
Vasco Duarte
Yes, it is definitely a mismatch.
- [26:21]
Goiko Atsic
And then kind of, I did a bit of experiments. How do we prevent this kind of proxying that's a total third party tool from thinking that my configuration files are text that need to be translated. And there's a bunch of tricks how to do that. And all of a sudden people can.
- [26:37]
Vasco Duarte
Now use this without translated proxy, which is.
- [26:42]
Goiko Atsic
And if you look at like what's, you know, at the moment, I still don't have fully internationalized interface because that costs a lot. And I can't afford to do it now, not in terms of money, but in terms of supporting it and things like that. But a huge, huge percentage of our users are using it. In Hindi, in Russian, in Urdu, in Vietnamese, without ever me doing it because they kind of benefiting from this ecosystem. We just stopped our software exploding and supporting it.
- [27:20]
Vasco Duarte
This is, I think, a great example of that serendipity that you have in the book. So check out the link to the 50% discount in the video description below. In the book Lizard Optimization you have this systematic process. So just as an intro for those watching this session and perhaps wanting to be more curious about it, what is the high level process that you describe in the book?
- [27:45]
Goiko Atsic
The high level process basically has four steps and it's easy to remember the steps because they start with lzrd, like Lizard One is to learn about how people are misusing your system, learn that people are using it in a translated context, learn that people are creating blank videos, learn that people are uploading subtitle files, learn what's unexpected from how people are using your system. Kind of the. From a technical perspective that would be exception tracking or you know, cache analytics and things like that. But we're not Talking about the technical stuff, we're talking about the software, the product itself working technically okay, just users are using it in an unexpected context. So learn about that. That's the first step and being kind of smart about how do we get that. The second one is to kind of extract the signal from the noise. So kind of the zeroing in on one behavior change. Zeroing in on one thing we want to look at like ok, at this stage we are improving stickiness, therefore we want to support people in different ways of using it. Or in this stage we're improving revenue or in this stage we're doing something. So zero in on one behavior change you want to do and then use that behavior change as a measurement of whether you're doing the right things or not. It's not whether I've supported translation, it's whether people are actually creating now audio files in a non English context and things that. So measuring that we're solving a problem, not that we are kind of delivering a solution, then the third one is once we have those measurements in place, we know what we want to do. We have kind of, we've defined a problem we want to kind of do. Looking at what obstacles does the software put in place in front of our users and then removing those obstacles systematically. Because if people are misusing the software, then our software is kind of somehow putting obstacles in front of them. You know, like I'm forcing people to create a video and they just want an audio file or they want to create a synchronized audio, it's reading timestamps or what are the obstacles to use existence removing those obstacles and then measuring whether we've done the right thing. And kind of finally the D is detect unintended impacts. Because lizards by definition do not follow the same logic as you and I. They follow lizard logic. So what you think is a solution might actually not be the solution or it might cause something totally other thing to happen. Or it might like, you know, I fixed it by allowing subtitle files but now it's reading the timestamps or it's not. Or I'll give you another weird example. We had a bunch of people with who were not able to enter the VAT numbers of their companies in the purchase field. Like, you know, for you and me, you look, it says VAT number. You know what a VAT number is? You copy and paste your company VAT number. People were really struggling with that. Lizards. Not a huge percentage, but some. And as a kind of owner of this product, I, you know, I'm not Morally opposed to people giving me money. So I want to support that, you know, and if they're getting stuck now I'm using an American payment processor and the American payment processor kind of doesn't really care too much about EU VAT numbers and things like that. So they were trying to validate the VAT numbers, but when people don't put the country prefix, let's say somebody's doing, you know, a finished number and they just do 1, 2, 3, 4, 5 instead of fi 1, 2, 3, 4,. The error said invalid number. And then people thought it's a card number, so they were trying to fix the card number and a bunch of things. There's like all sorts of weird workarounds that people are doing. Like there's a huge percentage of people that were selecting Russia as a country because Russian VAT numbers don't have a numerical, don't have a letter prefix to be able to enter it without the kind of letter politics. And then I have to fix the countries and things like that. Now I can't do too much about their validation because it's theirs, it's a third party thing. But I can control which fields are shown on the form. So I thought, well, why not just remove the VAT number completely and then when they finish the payments, I'll ask them, what's your VAT number? I'll deal with all the cases and things like that. And I did that and I thought, well, you know, problem solved. And detecting unexpected impacts actually stopped me from losing a lot of money because I was measuring whether people are doing this. And I realized now with the VAT number removed, actually the number of payments went down. It didn't go up because people would go there. They need to enter a VAT number to get a company invoice. They didn't expect to be able to enter it after, before and they were not proceeding with payments. My logic is not the same as the user's logic, which is fine.
- [32:40]
Vasco Duarte
This is also a beautiful example of data telling you what's wrong as long as you're willing to listen. Because a lot of the times we just act from our logic. Even if it's not lizard, it doesn't matter because what wins all the time is data, not logic. Right. And the example you just gave, I do that a lot with the people that I coach and it's very common that people have a very clear idea of what will work, then they try it out and then the data tells them that actually it doesn't work like that. And Many still struggle, even though they're looking at the data, they still struggle to accept. But that's one of the hard things, I think, for people trying to use this framework is going to be to believe the actual data.
- [33:25]
Goiko Atsic
Absolutely. But if you look at, like, what's happening, you know, the industry has got a lot smarter last kind of 10 years about this. There's a wonderful, wonderful, wonderful book everybody should read called Trustworthy Online Controlled Experiments by Ron Kochavi. And Ron became famous by tracking this stuff. Exactly what we were talking about at Microsoft, where in 2009 he proved that only about one third of the things they did at Microsoft actually delivered value. And then people started, because he became famous in this community, people kept sending him data from other companies. And he published a book called Trustworthy Online Controlled Experiments a couple of years ago where he has data from Slack, from Google, from Netflix, from other. And so I don't have the book in front of me at the moment. People should look at this, but the order of magnitude of the numbers is kind of somewhere between 10 and 30%. So the best companies out there, two thirds of the time the stuff they do does not deliver value.
- [34:31]
Vasco Duarte
And that's assuming they measure it.
- [34:33]
Goiko Atsic
And that's assuming they measure it. And now when you look at this from a perspective, perspective of somebody who develops a product, what percentage of stuff that you've delivered last year actually delivered value? How do you even know did it deliver the value expect, but also cause some other unexpected impact somewhere else? Is it actually driving the success you want it to drive? And that's really, really kind of curious to think about. And I think for me, again, because at the same time, I developed this software, but I also operate it and I own it. It's really important that the users are happy. It's really important that the people are kind of getting value from it. And I measure it. And that led me to kind of some really, really weird choices. But it's been a kind of wonderful success and a wonderful book.
- [35:28]
Vasco Duarte
Well, it's a book now, Lizard Optimization, not only is it useful for you, but it's. But it's going to be useful for all of us that can read the book. So when you think about the Lizard Optimization, the book, but I mean, the method now, what's the advice you have for teams out there, product owners or product managers that are thinking, I should really try to do this with my product. Where should they start? What's your advice?
- [35:52]
Goiko Atsic
Start monitoring predictable but unexpected stuff that users are doing. So we start monitoring kind of how people are misusing your software, Start figuring that out. There's a bunch of ways to do that. Session recorders are one. The other is kind of to get the technical teams to expand exception tracking, deviation monitoring and things like that that we already have for technical stuff expanded to kind of product monitoring and usage monitoring. A trivial way to start is any error screen you have in your system. Just make sure that when it's shown to the users, log the context of that, capture the context of that so you can see what kind of errors people are experiencing or weird stuff that people are doing, and then start extracting signals from that. That's a really, really good starting point.
- [36:45]
Vasco Duarte
We're getting close to the end, Goiko, and of course now we get to the recommended resources. One of them will be Lizard Optimization and the link is in the video description below. Another one is Trustworthy Online Controlled Experiments, which also has the link below. But what is a book, a video, a blog, a podcast that you would recommend for those interested in starting to explore the wonderful world of the lizard users.
- [37:10]
Goiko Atsic
So Ron Pohave's book is amazing. I collected a bunch of tools, both kind of technical and books and references and things like on lizardoptimization.org so people can view that. If there is another book to recommend for people to look at, it would be Itamar Gillard's book Evidence Guided. It's a wonderful product management book. Again, everybody should read it comes out of his experience managing Gmail. And then it's a big visible product and there's a lot of really, really good advice there, how to guide your product with evidence, how do we make it kind of data driven and things like that. And one of the things I love about his work is he has kind of this systematic confidence meter that's kind of when somebody comes with an idea, oh, this is the best idea in the world, then there's a way to kind of measure the confidence of that. And how do we increase the confidence in these ideas before we deliver them? So that would be another recommendation for anybody who's serious about products.
- [38:18]
Vasco Duarte
Absolutely. Anybody who's serious about product. That's a high bar to meet out there, friends. Let's do this. Let's read those books. Goyko. If people want to get in touch with you and maybe want to ask for your help, where should they go?
- [38:32]
Goiko Atsic
So goiko.net is very easy to reach me. Goiko.com also works. I have a benefit of having an unusual name, so kind of relatively easy to find online or if you just kind of Google for Goyko? Kind of. It's such a rare name. It's a very archaic name. I think the first person that comes up is this football player that sleeps with supermodels. That's obviously not me, if you see the picture. That's not me. But I'm kind of probably the second person showing with that name. So it's very easy to find online.
- [39:02]
Vasco Duarte
Yeah, I was gonna say that. Perhaps they'll find the football player. All right. And finally, at the end, we always end with one final parting word of advice for our intrepid viewers out there.
- [39:17]
Goiko Atsic
Look for the unexpected.
- [39:20]
Vasco Duarte
Absolutely. Look for the unexpected. And of course, check out the book Lizard Optimization. The Link for the 50% discount is in the video description below, so make sure you check that. And Goiko, thank you very much for your generosity with your time and your knowledge.
- [39:35]
Goiko Atsic
Thank you for inviting me.
- [39:41]
Vasco Duarte
We really hope you liked our show. And if you did, why not rate this podcast on Stitcher or itunes. Share this podcast and let other Scrum masters know about this valuable resource for their work. Remember that sharing is caring.