← AWS Podcast
AWS Podcast
#712: The Frugal Architect w/Werner Vogels: Watch Duty Keeps it Simple to Save Lives
00:51:09
What started as CEO and founder John Mill’s need for better wildfire information grew into tech that
Loading summary
Transcript93 lines
- [00:00]
Simon
This is episode 712 of the AWS.
- [00:03]
John Mills
Podcast, released on March 17, 2025.
- [00:10]
Werner Vogels
Hello everyone and welcome back to the AWS Podcast. Simon here with you with a very special episode. This is another of our really interesting series called the Frugal Architect. And there is no frugal architecture without that man, you know. Well, Dr. Verna Vogel, CTO of Amazon.com welcome back, Werner.
- [00:27]
Simon
Thanks, Simon. Thanks for having me.
- [00:29]
Werner Vogels
Always a pleasure. And we have some special guests here today for what promises to be a pretty engaging conversation. Firstly, the CEO of WatchDuty, John Mills. G'day, John.
- [00:42]
John Mills
Thanks for having me, Simon. Excited to be here.
- [00:44]
Werner Vogels
Good to have you here. And of course, wouldn't be a frugal architecture discussion without the cto, David Merrick. G'day, David. How you going?
- [00:51]
David Merrick
Hi, Nice to meet you both and thanks for having me.
- [00:54]
Werner Vogels
Awesome to have you guys here. Now, we are talking about a topic that is interesting to people around the world because it's about scale, it's about safety, it's about doing things on the smell of an oily rag, as we would say, doing things frugally and simply, but also things that are critical. And often people conflate criticality with expense. But often criticality breeds innovation. And so, John, maybe let's start at the start, if we can, of what is watch duty and how did it emerge?
- [01:26]
John Mills
Yeah, I mean, simply put, watch duty is a. Is an emergency alerting platform similar to what the government rings your phone at that very loud noise at random hours of the night. We're an organization that is providing emergency alerts regarding wildfire. And it was born, unfortunately, out of fire. I live off the grid in Sonoma county, where we experience this at a regular cadence. And after a couple of times you start to realize, like, who's going to tell me when danger is nearby and at what, what interval level? And after a while of living through it, it just decided to do something about it.
- [02:04]
Simon
So who are your customers, John?
- [02:08]
John Mills
Well, our customers really are everybody. It, you know, started out as a application for citizens because we assumed that the government themselves knew where everything was at all times. And as we've scaled, we found out that tanker pilots, dozer operators, emergency managers, firefighters, and everybody used it.
- [02:31]
Werner Vogels
So is that classic case of you built something thinking it was going to be used one way. But you know, customers get a choice, don't they? They choose what they like and what they want to do with things. And this went well beyond what you planned?
- [02:43]
John Mills
Yeah, I mean, it definitely is used for the same use case Just a different user base than we imagined. We didn't realize it was literally everybody.
- [02:52]
Simon
So I understand it started off as a volunteer effort. Can you tell us a bit about that?
- [02:59]
John Mills
Yeah. You know, Dave was a volunteer here, and so was I. Right. But Dave and I were working on it for almost, what, 18 months. Dave, I think. And then you came on board after that. Um, but it's still a lot. Lot of volunteers. Yeah.
- [03:13]
David Merrick
I was going to say it was two. Two full fire seasons. I think of them in fire season years, really. I guess it was less than 24 months. I mean, it was such an interesting start because I've never, you know, I've done, you know, hackathons and sort of four good projects before, but this one so quickly caught momentum and so quickly had a fit in the market. You know, you talk about customers and competition. The weirdest thing here is that it's. It filled such a gap that there was no competition. You know, like, no one else is doing this and no one else. It's not like we were an incremental improvement upon something that existed. It was, okay, you can go on Twitter and Facebook and have 12 tabs open for your life and safety, or you can use a simple app where, you know, experts are synthesizing this for you. It's like it's. It's such a. It was such a gap in the market that there really isn't competition to a degree, like in the classic sense.
- [04:12]
Werner Vogels
Well, I think it's interesting because it's one of those use cases, because we, of course, we don't call them wildfires here. We call them bushfires in Australia, and we're very familiar with those. But it's a classic seasonal thing that happens. And so for most applications, seasonality is not a good business model, because particularly when it's. Seasonality is not. It's really busy because it's gift buying season now. It's just normal buying. It's fire, no fire, fire, no fire. And the no fire time is where people sort of lose attention. And like you say, when I look at your app and the capability of it, it's like, wow, how didn't someone already have this? And it sounds like you were in that situation where you sat there, but you took that next step of action of saying, well, how does this not exist and how do we make it happen? What helped you make that jump? Like, what was it that, you know, we. We can all sit and go, well, someone should build that. But you built it.
- [05:05]
John Mills
Yeah, I mean, I. You know, again, after Experiencing a bunch of fires, two of them being a quarter, you know, quarter mile from. From my edge of my property. You start to realize that, like, someone has to do something about it. Like, I live off the grid, so my nearest neighbor is about a mile away from me. The nearest fire department is, you know, six miles, I guess. And so you start to realize when you make your own power, you make your own water, you live on your own infrastructure, like you're on your own. And so I could sit there and say, well, who's going to do this for me? Or I could start doing something about it. And luckily Dave was crazy enough to join me early and help me architect this thing.
- [05:43]
Simon
So tell me a bit about sort of what, how does I mean in Australia, Simon, tell me if I'm wrong. There is a user app where individuals can actually signal, oh, there's a fire here, there's a bushfire here, or it's starting here, but that's not what you guys are doing. And Dave, this is much more. We take professional information coming into the app.
- [06:12]
John Mills
Yeah, it's really more expert sourced than crowdsourced. Right. So like you, you can go on Citizen Next Door, Facebook and start screaming in the ether and tell someone that you see a fire, but is anyone going to hear it? You know what I mean? For us, it really starts with someone calling 91 1. And then we hear those dispatch signals. Our volunteers and some paid staff turn the radios on and start listening. So they know what to listen for, they know what these words and terms mean. And they're really synthesizing this information on the fly to make sure it's not literally the boy who cried wolf.
- [06:47]
Simon
So they are professionals or former professionals. They know how to interpret the signals that they're hearing on. On the radios.
- [06:56]
John Mills
Yeah, it's kind of a mixed bag, right. Some of them are kids who grew up with a scanner in their house because their dad was a firefighter. And they're literally worried, is my dad going to come home or not? And then some of them are the dads who've been fighting fire for 45 years and now retired and they can't turn the radio off. And so, like, once fire is in your blood, these people can't stop. It's a life of service.
- [07:18]
Werner Vogels
So let's, let's talk briefly about, I guess, what you reached for when you went to build this and how big is the team building this, this piece of software?
- [07:27]
David Merrick
I reached for the. The pieces that I knew. Well, you know, I went through the classic. All right, well, Let me draw up a diagram, let me write some things down. I wrote all these options for where are we going to host, what kind of database are we going to use? And my mind drifted. Like any shiny object engineer to like would be nice to learn a new thing. And then I thought about it a little more and I realized that was a terrible idea. You know, maybe, maybe if it was my full time job, I would have had the time to invest in, you know, learning a new language or architecture or database or really anything new. So everything, every decision ended up coming back down to things that I was really, really familiar with and that generally were very, very simple, boring choices. You know, I think a lot of people agree now that like you only get to spend so many innovation tokens, whatever you want to call it, you know, you get to pick a couple things that are wild or innovative or novel. But in this case it was like, well, everyone has a job, everyone's doing something else. We need to pick the absolute simplest things. So we're going to pick, you know, the technology stack I was using at my current job and you know, all the things that were already just the.
- [08:42]
Simon
Easiest path at the, at the beginning of it. You must have been thinking about, this is a volunteer effort, but how much is this going to cost me out of pocket myself? Did that drive any of your early decisions as well?
- [08:56]
John Mills
I was the only one who didn't have a job at the time, so I was, I was writing, this was my job, I was writing a lot of code. And so, you know, we picked Django, we picked Python, Python, we picked React and things that we knew we could pick up off the street to find volunteers who could do this, rather than, to Dave's point, the newest shiny, whatever, be super cool. But like, guess what, it's me in the woods by myself coding while Dave and others are volunteering at the time. And so it wasn't going to work very well if we didn't pick standard, you know, standard practice types of things including architecture like AWS and others.
- [09:33]
David Merrick
And then just to go to the cost, you know, it's easy to look back now and say, wow, 9 million users in the first two weeks of January. Good thing we picked cost effective choices. You know, we didn't know at that time what it would look like. You know, it was amazing how much pickup it had that first year. But we ended that first year with just shy of a hundred thousand downloads, I think. And that seemed crazy, you know, still not that expensive to serve in what we're doing. Mostly Read, read. Heavy, you know, API traffic. But, you know, we didn't even quite realize at the beginning what the scale of that cost might actually look like down the road. It was more like, if we actually get there, we can fix it then.
- [10:17]
Werner Vogels
And this is often a classic case of, you know, I'm building something to prove it works, and then I'm wildly successful. But often the timing of the wildly successful and the building it and then having the time to make it better doesn't line up. And particularly in your case, where, I mean, this thing went ballistic in terms of adoption, I think it was one of the top, if not the top downloaded app for a number of months against some of the names you'd think would be the most typical ones that are always downloaded. So you're sitting in this situation where it's a small team and we need to get into the composition of your team, because I think that's fascinating as well. But you've got this small team that suddenly doesn't want to have to be in complete, dare I say it, firefighting mode on the app. You just need the app to run. So having the elasticity is. I mean, that's. This is when it makes all the difference, doesn't it?
- [11:05]
David Merrick
Yeah. And I think, you know, we did understand that we needed. We needed to be able to pull the slider to the. To the right if we needed. And, you know, if we. If we really needed to spend money, we could all just find, you know, the cost to do so. But again, I think it. We also knew that there would be time. So you can go back to the seasonality, normally a terrible thing in software businesses. But it was actually, especially in the first few years, it was a. It was kind of like the dream on the engineering side, because you actually got periods of time where, well, we had no product organization, we had no sales, no one was breathing down our neck to do anything besides to improve the application. So you actually had these slow periods where you got to fix things, where you got to focus on performance, where you got to do that work that engineers fight so hard to, like, slot into the. Into the roadmap. And it was like, well, here's a couple months where at the end of these months, we. All of our libraries should be upgraded, all of our performance hotspots should be dealt with. All of that technical debt that you naturally accrue with a small team with quick decisions. We actually had those periods of time to really focus on it that's slowly going away. But I still think we're Trying to keep it in our, in our kind of approach and ethos because it really does help with long term stability.
- [12:30]
Simon
Now in that sense there's a. Well, I'm not saying it's as explosive as yours is, but retail in some sense, if you think about Black Friday, the month before Christmas, January is a. January, February, March becomes a time to fix things. Yeah. Because. But unfortunately in retail things never go back to where they were before. So. Yeah, you need to make sure you've.
- [12:58]
Werner Vogels
Got a significant problem.
- [13:01]
Simon
Absolutely. Yeah.
- [13:02]
Werner Vogels
It's not, it's not a good thing. Tell, tell us a little bit about the team. How big is the team? What are some of the talents on the team that, that make this possible?
- [13:10]
David Merrick
Yeah, I mean, you know, I, I don't actually remember how many people helped volunteer. I would say in the order of 15 that first year on the engineering side, 15 to 20 volunteers, varying levels of effort. Maybe like five to 10, like really core contributors that first year on the, on the volunteer side and that second year, probably the same group. And then, you know, it's a very unique composition because we started out fully volunteer for the first, let's say two years. And then we switched to a model where well, I joined full time, you know, CTO of one, you know, quite the title, and then hired a contractor as I realized I needed a little extra help. And really that whole first year was just, just myself until we started hiring a little bit more that second year. And now we're at seven, seven engineers. So still the one, the two piece of team. The nice thing for us right now is because we had such a strong group of contributors, we were able to recruit some really, really talented engineers from, from that group, which is quite the, quite the interview process. You know, if you've worked with someone and you know, in terms of how we're composing our team right now, you know, we still don't really know all the things we want to do, so we've been just trying to hire really, you know, that's an overused word, but full stack engineers that can are really focused on problem solving. You know, I don't think we don't have a lot of novel technology that we're building, but we are solving very, very concrete and real problems. So we're trying to hire and get engineers that love to own a problem from the beginning to the end. You know, there's not going to be someone in product building out all these requirements and then a deep technical dive. It's like you need to be the person that loves Seeing an issue and then seeing it all the way through. And the nice thing is right now we have a really, really experienced team, a lot of talent and density. So we're able to get a lot done with a really small team.
- [15:23]
Simon
How did originally your volunteers show up? Were they people you knew? Were they early users of an app? I mean, how did you manage to get five people on board to donate their time? It's a problem that most nonprofits have where they only have two people that are maybe paid but rely on a lot of volunteers. How do you find the people that are willing to donate their time to you?
- [15:50]
John Mills
Yeah, I mean, we found them pretty early on through, mostly through our networks, through mine and Dave's. I mean, a lot of the people that we brought on board were friends of ours. Dave and I worked together at multiple companies. I have a couple other friends who I've worked with or known for a long time who cared about this problem. We told a good story. We had a good product. I mean, our first rev took. I worked day and night for almost 80 days. I built probably 80% of it. And then Dave and others built, did the stuff either AI couldn't do or I had to sleep at some point. And then after a while, it picked up, people started to see it, they knew what it was, they were using it. And it makes it much easier to then recruit people because unlike a lot of nonprofits, they often build things for other people that they have never experienced those problems, which I think is really hard.
- [16:38]
David Merrick
I'm just going to add in that John is very good at convincing people to help them, so he's selling himself a little short. There's.
- [16:46]
Werner Vogels
Well, it's interesting too that you, you, you. You often call yourself sort of a tech first nonprofit, which is kind of a different profile than. Than we're used to. Help us, I guess, nailing onto that and, and what it means to have now here we have to have a CTO and a, A prior CTO running this thing. How does it affect the culture of the organization?
- [17:09]
John Mills
Well, it really comes down to product. So it has nothing to do with the tech stack. Right? That's first. First and foremost. Like, who cares if it's peanut butter and jelly, duct tape and bailing wire? Now, can it take 100,000 requests a second like ours just did with a tiny team? Fortunately, I could, thanks to AWS and Heroku and other providers who allowed that to happen. But you know, that said, I mean, this is a really organic product that is made to solve a problem. And so his first revision is essentially a simple newsfeed with, with a geospatial map. Right. None of this was all that complicated. So if you kind of like strip away everything you don't need and just give it what it's supposed to be like, it's a lot easier to get something off the ground to think clearly and simply and realize we don't need complex engineering to solve these problems.
- [17:56]
Simon
Right.
- [17:57]
John Mills
We need a very elegant solution to a very annoying problem. And the answer was actually using human radio operators to solve it. The tech came second. And so there's a lot, quote unquote, better products than ours out there in terms of, I don't know how you want to measure things, better code or whatever it may be, but we have better outcomes, right? Outcomes are what matter. No one cares what it's written in. And so it really comes down to first principled product.
- [18:24]
David Merrick
I mean, I just want to add on. I think you're underselling our tech a little bit, John.
- [18:29]
John Mills
Early Jack, early Chuck. Pretty simple man.
- [18:32]
Werner Vogels
You can see John's become a CEO.
- [18:34]
David Merrick
You know, I think, I think, you know, you look, everybody, just like everybody in LA just downloaded Watch Duty and they got to see what Watch Duty looked like after four years of revisions. You know, I think that's the big difference between a Tech first nonprofit and a nonprofit that wants to do something like this. Because we set ourselves up to iterate very quickly to be able to take feedback from the SMEs, from our community, from ourselves, from everyone that has every single support ticket that's ever come in, we've looked at and reviewed. You know, we've probably, I don't actually know the number on this, but we've probably released our mobile applications upwards of, you know, upwards of 50 times a year. You know, that comes out to 200, 300 times we've released. You know, we're up in the thousands of PRs in all of our backend and front end repos, you know, the ability to iterate quickly and have a, I think a world class product on the tech side is that other difference because you can want an outcome and you can, and this is outside of the kind of scaling thing where 9 million people using your app doesn't tank it. But I think even on the product side we've been really focused on, on iterating and bringing you know, really, I think what a lot of people consider just normal product development in the sort of, in the startup and, and tech world to a user facing problem, you know, a public facing problem that a nonprofit is the best, best suited to just to solve and provide. You know, this wouldn't work if we're trying to charge people for this data. You know, we, we, even if we didn't believe that everyone should have it, I think we never would have gotten the traction. If we were trying to, to profiteer off of people's most, you know, panicked moments, maybe it would have gotten somewhere. But I think that's the big difference.
- [20:34]
Simon
So can we. So we talked about customers with, in essence, the users that, you know, if you normally would build a product, you would work from your customer backwards. Like they say, you care about the outcomes. But there is a massive input stream here as well, because without the input of the volunteers that are actually putting things into your system, this wouldn't work. So that. Do you consider them to be another set of customers? What do they need? What do we need to give them to be able to get this accurate stream into the application?
- [21:13]
John Mills
So I think stepping back is, you know, it's when we found these radio operators, they were using Facebook and Twitter, right? So they were using a very lackluster product to solve a very complex problem that was involving life and death. And so the bar was very low, right? So obviously, four years later, we built all sorts of advanced tooling and signals, noise processing, whether it's AI bots and whatever it was, but like, we were immediately better than what happened before them, right? And so we, we had leaps and bounds, better ideas about how they should be doing their job. Like, for example, they didn't even really communicate with each other. They kind of message each other in DM and Facebook or whatever it may be, but they kind of be running their own independent Facebook pages or Twitter pages, and they'd kind of know each other and like follow each other, but they never talked in real time and slack. So imagine like four people covering one fire, or like the Palisades fire, for example, which was like six to 10 people at a time. They're all collaborating in real time. So just the fact that we put them together in one place and gave them a community to talk and collaborate was a huge leg up, let alone all the systems that we built on top of it. And so again, boiling it down to like how things were versus how things going to be, it was, it was not as challenging as you may think. Now that said, as Dave's going to chime in, we spent a lot of time building tools, right? But ultimately it was so much better than it was that it became much easier to get them enticed in this and asking for features and ways for them to work better and faster.
- [22:53]
David Merrick
Yeah, I mean it's a unique organization. It is very much in one side, a product facing tech solution. But it would be nothing without that input stream you're talking about. And it's incredibly important. It's the secret sauce of what makes Watch Duty so compelling to everybody. And they are very much our customers in the other sense because, you know, there's only so many of them. It's hard to find these people. You know, it's a very specific domain expert that wants to spend their time and energy and. And really it's a passion for most of them. So we're trying to give them the absolute best tools that they can have, you know. You know, we had some volunteers that were also reporters on the engineering side. They were incredibly motivated to help fix these problems. We have advocates within their side that tell us, hey, this is busted. This we waste time with. Sometimes we just watch what the reporters do and say, you've been doing it that way for how many months? That's a lot of time wasted. You know, we had a. There's an anecdote last year where we found out that all the reporters had to refresh Facebook and Twitter pages all the time to find out how to report on more. Kind of like the end, the tail end of fires, you know, because that's unfortunately where a lot of public information is, is posted right now. So they just had like a panel of 20, 30 tabs and they'd go, every 10 minutes they'd go refresh every single one. And it was like, what you're doing is super valuable, but that is not valuable. We can help solve that problem. So give us a couple months next spring you won't have to deal with that anymore. Because that is definitely the hardest part of our scaling is the reporters and the contributors and that human element which is I think this challenge.
- [24:46]
Simon
The quality of data really relies on them much more than anything else. By the way, can we do one little sidestep? Because I'm actually just curious. In it you mentioned Geo, John. What I find over time traveling the world and go to many different places is that commercial maps are not that terribly accurate when it comes outside of cities and organizations like Humanitarian OpenStreetMap and the Open Street Map Group and things like that. What kind of mapping technology are you guys actually using? Because this is mostly off the grid, as you mentioned, John.
- [25:28]
John Mills
Yeah, it's a good question. I mean we've. God, we've done so much experimentation with, you know, different tile sets from Esri Mapbox, OpenStreetMaps. And honestly, we still struggle with it. You're. You're hitting the nail on the head is like, so out in the wildlands, as we call them, like, there's oftentimes, like, satellite passes that are two years old, and you're like, that house burned down, that water tank's not there. Like, that's a dirt road that's now closed off. Someone bought that land and it's gated, and they have no idea. And so we still struggle with that, honestly. And we're trying to find answers to those, to those problems. And we're definitely not doing Google Streetcars ourselves or, you know, helping open Street Maps, although we wish we could. We have no way of doing that. And so we're trying to think of other clever ways of doing that, whether it's buying satellite imagery and partnering with those organizations. But there's a lot. There's a lot to be gleaned by the Earth observation community, which still hasn't seemed to solve this problem either. And I scratched my head living out here. I'm like, how did I just get here? And I'm seeing all these issues, and we've been able to open the can of worms and solve a bunch of them, but there's more underneath them. And this is a big one that we don't have a clear answer to yet.
- [26:45]
David Merrick
I mean, it's. It's a big challenge. But I think, you know, part of the challenge and constraints that we have is we can't bite off all these problems. You know, we get a lot of user reports around. You know, this street is named wrong. This street isn't there. This satellite pass is old. And, like, it takes a lot of discipline to be like, we're not going to build a middle layer where we start overriding OSM data. Like, we have to be really focused on our core competency and sometimes just be like, well, you know, I wish that our product could be a little better on this aspect. I wish we could do it. But, like, we're a team of one, we're a team of three, we're a team of five. Like, it's enough. We could work all day and night and just distract ourselves from the actual mission. It's frustrating to have the street behind you mislabeled, but it's more frustrating to not have the product that we brought.
- [27:38]
Simon
You because it is a massive problem. I mean, India is a really great example where, you know, many of the roads are not either. Not you can't drive a big truck over it or a Google Street Map fan or whatever. So there's this one app that actually is for Tuk Tuk drivers to connect you with a tuk tuk driver and they in general go out into the woods. So this app now also actually tracks that data. Where did the doctor go? So now we kind of know that there is a path to that. Which brings me back to actually another question for you guys. You collect a lot of data, you get a lot of input signals from all these professionals. Is there something that you do, let's say, with that data after the fire? Because this must be an extremely important data set that is also for firefighters. Very important.
- [28:35]
John Mills
It's really useful for after action reports, which is what the fire department will do after, you know, the dust is settled and the last engine leaves. But a lot of our data, like believe it or not, is publicly available, right? Like, what's really unique about us is our news feed, which is the radio reports. But again, when it's all said and done, that's all public as well. This is all foiable public information. So a lot of it can be had. We just have kind of a common API or endpoint to get some of that data. But I found that like, for example, like, we don't know that like this part of the fire was crowning, meaning the, the canopy was on fire.
- [29:17]
Werner Vogels
Right?
- [29:17]
John Mills
We don't know that this was the hot spot, this was the cold spot. And like the little subtle details that people want to know, like we don't know either. We might have one of the better data sets in the world, but you'd be surprised how finite it actually is. And I remind people that like, although we, we may be the, the best quote unquote, we still don't have what we need because we don't know where the perimeter is as the fire is moving, his throwing spots or embers half a mile, a mile ahead of itself, lighting fires ahead of it, pushing the fire with the wind. Like we don't know that. No one knows that. Right? And so it's really knowing what I know now. There's still so much more that we don't know. And it's really quite interesting how challenging this problem is. And until we get really high resolution satellites doing 1 meter resolution at very, very sensitive heat detections, we're not going to know all that information, to be honest with you.
- [30:17]
Werner Vogels
Do you see a world where things like drones that are, I guess, dedicated to seeking out fires and changing our fires and telemetry back and sort of AI driven, trending, et cetera. Like what, what's, what's the future looking like as you sort of start to reach into what could be versus what you've achieved so far?
- [30:39]
John Mills
Yeah, I mean I think it's interesting to think about. You know, we like to think about, skate to the replac is going right and think about the future. And so there's certain, like there's ways to cut this and really think about it. So for example, let's use the Palisades fire. The Palisades fire was a hurricane with fire inside of it. So if we found it two seconds earlier, not much we can do about it. Right. Unfortunately, like the reality is, and people don't want to hear this is that like when you're fighting a wind driven fire, throwing embers across the head of the fire at that kind of speed, at that rate, there's, there's not enough AI and drones in the world to stop it. Right. That's the reality now where the AI and the drones come in handy and let's call it, I forget AI. I don't think that's irrelevant for this conversation. Like detection, right. Is really what we're discussing. Like there are great examples like the fire that happened to me that got me in this business. There was a lightning strike 12 hours earlier, that was a smoldering tree that didn't matter until 12 hours later and the wind picked up, set those embers flying. Next thing you know there's a 50,000 acre fire. That would have made a difference if we had known that was there, that tree was on fire, we absolutely could have stopped a mega fire from happening. And so being able to detect this stuff and respond to it, whether you're having drones dropping water, whatever it may be, and suppress these things in the middle of nowhere, that is definitely something that will stop some fires. Sadly not all. And so it's kind of a, let's throw everything we have at it, but realizing that like 100% never going to happen again. It's just not a reality we live in.
- [32:16]
Werner Vogels
Nature is nature and nature is awesome in the good way and the bad way. But it's interesting, you know that that lightning example is a great one where it's a very small incident and I'm sure there's hundreds of lightning strikes during certain times of the season around the area. And it was just that one that caused a disaster. It can be that little, which is I think the other interesting part of all the data you're bringing in. So you've got open data sets, you've got input from volunteers from reporters, et cetera. Like, this is not just sort of structured and unstructured data. This is data of all kinds of dimension here. How do you unpack that in your, in your brain? How do you, David, how do you sort of tackle that from a data modeling standpoint at least?
- [33:02]
David Merrick
I mean, I think right now our best answer is that it's not that the reporters are an input stream, is that we're giving the input streams to the reporters. So our best solution right now is to give them the subject matter experts, the cleanest, fastest information and to reduce all the other friction that they would normally encounter when trying to do this reporting. You know, they're able to do a great job of synthesizing because, you know, there's a lot of details that they hear on the radio about the real time firefighting efforts. Not all of that is relevant to the end user and their job. One of their jobs is to make sure that what they're reporting is the most relevant information. You know, you don't want to get your phone pinging you every two minutes saying like, one more engine got added, two more engines got added. You know, it doesn't, it's not, it's not, it's not actionable. You know, I think what they do such a good job at is finding that balance between giving people information that is actionable and not leaving them scratching their heads saying like, oh God, what's happening? I haven't heard anything in six hours and I live two miles from this fire. So we really view it as what's the, what are the best systems that we can give These ex human experts, they're going to do the synthesis and analysis and then out comes actionable information for the public.
- [34:28]
Simon
Human AI, which works exactly better.
- [34:31]
John Mills
That's right.
- [34:31]
Simon
It's a lot better than. I mean, the. Absolutely not.
- [34:36]
David Merrick
I mean, even when we do have automation, even when we do have automation, it's always human in the loop. You know, I think that that is really overlooked a lot in terms of like, the value of where AI and automation and technology can go. Because you can get so far by just giving people a review button as opposed to making them do all the work or being okay with failures or error. And in our situation, we're very conservative in terms of ensuring that the accuracy of the data is very, very high. Going out, I was gonna say, I.
- [35:08]
Werner Vogels
Think that philosophy of needing to be right and valuing the correctness versus the speed is such a fundamental tenet in a safety critical world that's often overlooked.
- [35:19]
John Mills
Yeah, there's a signal that I want to tell you guys about since this is a geeky podcast and Dave's probably gonna know what it's gonna be, but this has gotta be favorite thing that we do. So fires are fought with radios, right? Shovels sweat. And radio is how fires are fought. And there's a lot of intelligence on those, on those analog radio channels, right? 150 ish megahertz depending on where you are. And so what's really interesting about the fire service and actually all emergency services is they put digital signals on an analog radio network. And so what you'll hear by listening to the first responders is you'll hear DTMF tones, right? The same thing that your phone used to produce or the fake ones that the iPhone produces that doesn't do anything anymore. Right. That sound is something that they actually use to communicate with each other. And it goes extraordinarily deep. So here's what happens. There's someone at a dispatch office who's going to order an engine and they say, hey, I need someone to go investigate a fire. Someone called in a fire. 1, 2, 3 Main street, hang up the phone, some dispatcher will radio in and they'll often use CAD systems and whomever. But anyway, what'll happen is they will tone out is what it's called a firehouse. And so let's say that tone is 1, 2, 3, 4. So on that radio frequency, you're going to hear beep, boop, beep, boop. There's a number, right? That number means something in that county, but two counties over, that might be a bulldozer or that might be a battalion chief. Well, our team are those types of people who decided to listen to all of them and begin to categorize them all. And so now we do DTMF tone demodulation and we built a yellow pages out of the analog network to turn into a digital signal to then drop it into slack to say that someone ordered three bulldozers in this general area. So that's the length that we've gone to solve a complex problem.
- [37:11]
Werner Vogels
When we say diving deep, that's diving deep. I love it. I love it.
- [37:14]
Simon
Okay, Given that it's also called the Fugal Podcast. So let's go back to money a little bit. You were just a bunch of geeks building something that you thought was going to be extremely useful for yourself, that at some moment, you know you be. You understand that you need finance for. For this. And, you know, you made a decision to become a nonprofit organization. So that means you start relying on donations or at some moment also, you have subscriptions. So how does that, first of all, how does the process work for. Work for you?
- [37:55]
John Mills
Well, the interesting part is we were a nonprofit from the beginning. And I actually formed a nonprofit before Watch Duty. And the nonprofit's mission statement is solving obvious problems for underserved communities. So I got that paperwork approval in April of 21. We started to formulate the idea earlier than that. We started building code on May 17th of 2021. And so we actually have always been a nonprofit. We were still a bunch of geeks trying to change the world, but that hasn't changed, and neither has the nonprofit status. So it's always been that way. And then ultimately, to Dave's point, like, the end of our first year, we were like 100,000 users. It's like a drop in the bucket. And so we got donations from Amazon and Heroku. Early on, we didn't really have any bills. Right. So we publish a lot of this publicly. You can see it on our annual report. But, like, we spent no money the first year at all. It was just human intelligence for the entire, you know, entire first rendition of this project.
- [38:58]
David Merrick
I mean, it's also easy to spend no money when you have five engineers giving their time. Uh, I mean, yeah, like, the, you know, that was the output. That was. That was the input was like, you know, we had donations from some cloud providers and. And people donated their time. Um, I think the transition from a full volunteer to, you know, the paid staff and, like, all right, we're really growing up here. That was where we, you know, started to figure out, all right, well, what is our business model going to be? How are we going to run this? We don't want to throw galas. We don't want to do the traditional nonprofit grant funding cycle where we're beholden to all this. So we kind of went to what we knew, which was treated like a startup. You know, most of it's going to be salaries. How do we get enough money to kick that off? And then it was never going to be a problem about user growth, it seemed like. Or engagement. You know, we. I think, what was our NPS the first time we measured it? Like, 92.
- [39:53]
John Mills
You know, like, paid was in the 90s. Yeah.
- [39:56]
David Merrick
Yeah.
- [39:56]
John Mills
Dave was 86.
- [39:57]
Simon
People really like climate change. Your. Your application area is probably only going to grow over time.
- [40:04]
David Merrick
Well, sadly, a growth Industry. Yes.
- [40:08]
John Mills
So I think after the first 18 months is when is when I put a first million dollars into the company. And that's when I was able to call Dave and say, hey man, remember that thing we're doing? Well, this is a real job now. And so, so I kick started it with a million bucks. We hired Dave and some other help at the time. I can't remember. The years blend together. There is no fire season anymore. It's just fire all the time. But anyhow, we were able to then hire top tier talent. It wasn't just volunteers anymore. It was volunteer supported operations. Like any nonprofit, they have volunteers and they have staff. Right? And so we were able to kick that off and get that running. And then to Dave's point, like, you know, we knew that we, we could make a self sustaining business as a nonprofit. And it actually took me at least a year or so to figure this out because every lawyer, CPA nonprofit I talked to told me I can't do what I'm doing. And then I started to dig in deeper. I was like, well what about aarp? They're a billion and a half dollar a year nonprofit and they sell services, I. E. A magazine to the elderly. And then they get discounts at Safeway. And turns out then the lawyers are like, oh well, you can do that. And I'm like, well, what's the difference of what I was telling you, right? They're like, well the answer is they're not paid to think that way. Right? And so it took a lot of like thinking about this problem, like unlock it. We're like, wait a second. Museums and the AARP work the same. Send with NPR sends you a tote bag for, for donating, right? We just give you extra features in the app. And so the beauty is of nonprofit is that as long as you're selling, you know, services that are part of your program, which is what the money goes to, then the proceeds coming off of that are nonprofit dollars. Non tax deductible. Do not get taxed on revenue. And now we operate like any other of these nonprofits does. And that was the big unlock for us to figure out how we make an actual business out of this thing.
- [42:05]
Simon
So if you think about the architecture side of it, when you started offering, let's say for pay or you know, for pay features, did that change the way that you thought about your architecture? Because now suddenly you have a reliable income stream.
- [42:22]
David Merrick
Yeah, I mean, I think, you know, the, the marginal cost of all of those was, was really quite low. So it didn't have a huge impact on like the spend. You know, it started out as a very small percentage of our users. So it wasn't. And the percentage of users that donate and become members is still under two and a half percent. So you know, we're giving away this data and we always will for the life and safety stuff. And, but that's to 97% of the people using the application. So it's like we, we are, we always had to build for the scale of people that are not going to pay us anything. So it was always how can we make sure that this is going to be done efficiently? We need to be able to have buffers. We need to be able to. When it was really more about like what does it look like when it doubles triples 10 x's on the cost side. And then we'll find out how, how the income will, will get to us. Now in terms of, I would say the biggest difference about having a more reliable income stream is they're able to hire. You know, it was like, you know, we, we, we had a lot of confidence that this was a market fit, that this was incredibly useful. But we can't, you know, there wasn't a VC fund that was going to, you know, continue to funnel us money until that finally panned out. So we needed to make sure that we were hiring responsibly and effectively for kind of like I guess a bit more conservatively than I think a traditional for profit business would have. So it really helped us understand how we could grow that team out.
- [44:00]
Simon
Do we, I mean your origin and your focus is really. And also of course the emergency of the past, let's say six months at least is fires, fires in California. But I can imagine that this application would be equally useful for hurricanes in the south of the US or other type of flare up emergencies. Like to our. Do you, do you guys see interest into that? Not, not necessarily from you, but do you get contacted by other people saying hey, we would like to use it for this particular application area?
- [44:38]
John Mills
Yeah, I mean just like, just like we started as a nonprofit, we also didn't put fire in the name of the company on purpose. Right. So first of all we're in 22 states, not just one. Right. So California happens to have 38 million people which is like most of the population of the west. But, but we've been covering fires in many other states for quite some time and now we're all the way up to the Mississippi river minus Louisiana. So we're already pushing eastward and then we're now launching our, you know, investigation and some new features out soon. Ish. We're trying to figure that out regarding water issues right now. Water happens to show up in very funny ways. It can show up at a tsunami, it can show up in a river, it can show up in a hurricane. And water kills more people than fire actually.
- [45:29]
Simon
Right.
- [45:29]
John Mills
People run from fires, but they drive their Prius into the river thinking they can forge it. Right. And it's a strange human phenomenon. Right. Of like kind of a misnomer of how powerful water actually is. So it's a great question. And we're going to continue to explore what that looks like and how we deal with these other disasters that are geospatial in nature, that are temporal. And unfortunately, we've seen many issues and other problems. Right. We just had a tsunami warning that went out to a lot of the American west on the coast, and that warning said run to higher ground. Now, some counties and areas went to extremes, like I have friends in Berkeley that all packed their Subarus and drove up the hill and they found out that 25 minutes later there was no tsunami, it was not going to make its way under the Golden Gate Bridge and it was not going to wipe out Berkeley. Meanwhile, they're panicking for 25 minutes while a tiny little swell, perfect for surfing, went under the Golden Gate Bridge and. And did nothing. Right. And so this problem exists everywhere because we don't think about borders and boundaries, right? Like, we look at things from 80,000ft down and say, like, what do the people need to do in response to this geospatial problem? And how do we stop pretending that these jurisdictions actually make any sense? Because they don't. Right. These disasters take from us all equally and everyone deserves equal information. And so we try and think about it from a much broader perspective than just fire. Just this county, just this area, or just this disaster.
- [47:07]
Simon
Yeah. Here in Europe, one of the parts in Europe the biggest deal is indeed floods. The big rivers like the Rhine and Maize and others are all more and more. The dikes haven't been built as high as that they should have been 20 years ago. And it's a massive problem here as well. And a lot of reporting is inaccurate around that at the moment. So I can see many different areas where you guys could be successful.
- [47:38]
Werner Vogels
Definitely more work to be done just.
- [47:40]
David Merrick
To add on there. I mean, I think what we've seen is that I don't think this takes anyone by surprise, but the government is not the best at building agile Software, best of intentions, but it's just not their strong suit. And there needs to be these really high class, well thought through, product driven solutions to really pressing problems and they're not being built. You know, I don't think what we're doing with Wildfire is going to be the solution for a hurricane. You know, there's some really unique characteristics about the, the temporality. As John said with Wildfire, it's probably closer to a tornado than anything else. Except tornado is more just like hunker down, get in the basement. There's not a lot of other action you get to do. So, you know, we're not sure what it looks like in everywhere else, but we do know that there's this big gap. The gap is high quality products delivered for free to the public across jurisdictions and boundaries and borders. And I think that, that there, like you said, there's a lot of room to grow there, but we really have to figure out where we fit in. You know, what we're doing for Fire may not be the product fit, the market fit in every disaster, but it's going to be there. And we're working and investigating what capabilities we can add to our platform to fill those, to fill those gaps.
- [49:02]
Werner Vogels
Sounds like there's plenty to get on with, which is fantastic. John and David, thanks so much for joining us today on the podcast.
- [49:08]
John Mills
Thanks for having us.
- [49:09]
Simon
Think Torment. Brilliant, brilliant product and absolutely essential service.
- [49:15]
Werner Vogels
Absolutely. And Werner, as always, I think this kind of story again emphasizes that frugality in all its forms is an important thing for architects to think about. It's not just money, it's time, it's effort, it's problem fit. There's a lot to unpack here.
- [49:32]
Simon
Well, and team size for example, in indeed. I remember Dave said at the beginning, you know, your, your tech stack is mostly also determined by who, who's willing to come on board, you know, and indeed, if you Django and, and, and Python, you will be able to find a lot more engineers that are able to continue. Yeah, we actually in the early days of aws, built one or two products on the perfect programming language, Erlang. Don't even get me started. It was not my idea.
- [50:09]
Werner Vogels
We're not having this conversation.
- [50:10]
Simon
It's not my idea. However, what that team pretty quickly figured out is that they couldn't move anywhere else in the organization because that's what you can do within aws. You know, you work for two years on that particular database, you want to do something in robotics, you go to the other side. Nobody could leave on that team because you couldn't hire new Erlang programmers. So course of action after two years was to rewrite it in Java. Not because that was the better choice in terms of technology, but there are so many more aspects to development than just is this the absolute best new best particular technology for this particular thing? If you can't hire the people for remains a people problem.
- [50:57]
Werner Vogels
The right tool for the right job. Thanks everyone, and thanks everyone for listening. Would you love to get your feedback? AWS Podcast.com is the place to do it. And until next time, keep on building.