← Software Engineering Daily
Software Engineering Daily
Autonomous Drone Delivery at Scale
00:50:30
Autonomous drone delivery has long been the stuff of science fiction, but ongoing advances have moved the space from experimental to operational. Zipline is one of the leading companies in this space, with drones that charge between missions and fly au...
Loading summary
Transcript53 lines
- [00:01]
A
Autonomous drone delivery has long been the stuff of science fiction, but ongoing advances have moved the space from experimental to operational. Zipline is one of the leading companies in this space with drones that change between missions and fly autonomously to deliver packages directly to customers. Kyle Madonia is the VP of Application Software and IT at Zipline and she previously spent a decade as an engineer at SpaceX. In this episode, Kyle joins Gregor Van to discuss how Zipline software stack powers end to end autonomous delivery, the engineering challenges of managing drone fleets at scale, and how the team approaches software releases for safety critical systems. Gregor Vand is a security focused technologist, having previously been a CTO across cybersecurity, cyber insurance and general software engineering companies. He is based in Singapore and can be found via his profile at Van heu or on LinkedIn.
- [01:14]
B
Hello and welcome to Software Engineering Daily. My guest today is Kyle Madonia.
- [01:20]
C
Hi, it's great to be here. Thanks for having me.
- [01:22]
B
Yeah, awesome to have you here today, Kyle. We're going to be talking about all things autonomous flying, which I don't think we've ever had an episode yet on this, on Software Engineering Engineering Daily. So you are at the company Zipline, which we're going to get into in a second as we like to do on Software Engineering Daily. I believe currently you're the VP of Application software and IT at Zipline. You do have quite an interesting career path to getting to Zipline as well. I think that'd be interesting just to hear about and sort of how it influenced, I guess, you then joining Zipline.
- [01:56]
C
Yeah, my background is all computer science, so I studied computer science in undergrad grad school, started working as a software engineer in a few different companies and ended up joining SpaceX in 2013. It was a company that nobody had really heard of at that point. I hadn't really heard of it and I thought, oh, it's aerospace, it must be boring and slow. Why would I want to go there? And then I visited the factory and I was clearly wrong and thought, well, this was a mission that I could really get behind. And I spent the next 10 years at SpaceX growing into leadership roles, but really focused around building out the internal systems that we use there. So I built out Warp Drive, which is the enterprise resource planning ERP system that SpaceX used, that Tesla also ended up using. I built out a lot of the systems there. I built out the network operating center for Starlink, how we communicated with satellites, managed the fleet, the telemetry system that we had for that as well, even worked on all kinds of random internal systems. For example, in Starbase, we built out a leasing system, how we handled leasing properties to employees who were now moving to Starbase. It was just anything that SpaceX needed, we were building. And after about 10 years there, I started thinking, I wonder if now it's time to see what else is out there. And SpaceX said had grown so much and was so successful in so many ways that I wanted to see what else existed that was really exciting and have to be around this time that Zipline contacted me and I started looking at what Zipline was doing and seeing the mission that Zipline had where for the past around 8ish years, zipline was serving medical products in Africa and delivering blood supplies, vaccines, nutritional supplies, all of these to really help save lives there and to help improve the health system in Africa. I felt like that was already such an important mission, but now Zipline was at this inflection point of launching in the US and we now want to bring some of this to the urban areas. And really our goal is to be able to have a logistics systems that serves all people equally. And I knew leaving SpaceX like it had to be a mission that I really believed in because that's important to me. But I also wanted to be at this place where we weren't sure would it be successful, because I think that's always such an interesting place to be, where it's not when are we going to be successful, but it's if we're going to be. It was the same kind of thing in 2013 when I joined SpaceX. You didn't really know if we were going to be able to achieve this. And I saw that the areas that that Zipline really was growing and was something that I could help in. So I decided to make the jump and join the company. And that was about two years ago. And it's been a fun ride here now as well.
- [04:34]
B
Yeah, that's amazing. And I think it's very interesting how one huge mission effectively was taking humans off the planet. And like, how can we put them in a different place? And then this mission is all about humans on the planet, quite frankly. And I really like that.
- [04:50]
C
Yeah, that's actually what I was saying when I started just putting out some feelers was that I wasn't actually sure what mission was after because how do you say, changing our mission from making humankind multiplanetary. What is the next mission? I just want to do something here on earth. I've spent 10 years. How do we go and explore and be able to get off Earth. But now I want to say, how do we make things better for people on Earth? And Zipline just had the perfect combination where we were trying to make things better for people on Earth but using the skies. And I thought that was so key to being able to make such a huge impact.
- [05:22]
B
Definitely. So we're going to get into the end to end experience of what Zipline is shortly, but just more of like a super high level. You have touched on it. It is obviously you've mentioned deliveries, for example, but what is Zipline? Just to get us understanding that one.
- [05:37]
C
So Zipline is a company that does deliveries with autonomous drones. So we build and manufacture all of our drones here in the US and then we deliver a variety of products. And the important thing for us is that we were able to deliver extremely fast and safely to people for whatever reasons you need it for. So part of our mission is always in the healthcare space. That's really our roots in launching in Africa and internationally with what we've done with our longer range drone that we have and now with what we call our platform 2. But it's more of our high precision drone that we have built now launched in the US we're delivering retail products, food from restaurants, prescriptions. We still want to stay very close to the healthcare space. We're gonna have more healthcare partners coming online and really, again, it's about just delivering to people whatever they need. When we talk about why I joined, I think when I was talking to Zipline, I have two young kids and my kids, there was one night where my husband was out of town and both my kids were sick. And I'm like, I just need Tylenol. Like it had to be specifically Tylenol if you're a parent, you know. And I didn't know how to get it to the house because I needed it fast. And everything I was looking at was going to take so long. When you have two screaming children, like, you know, you cannot take a long time. It just happened to be the next week when Zipline contacted me and I thought like, that would have been my solution. And I think that's what we're seeing as we launch. It's like they're parents who are ordering formula for their babies, for medications for their kids, the elderly who have a harder time leaving. So that's where Ziplines is trying to solve these problems of just delivering quickly for whoever needs it.
- [07:13]
B
Yeah. And I imagine you've talked how the core market at the moment, I believe is in Rwanda and Ghana. For example, and what you've just mentioned, that's like amplified so much. In these countries, these regions, there is no, let's just take a name, Uber Delivery for X. There's none of that. Right. And so this is the only version that's not like a car having to traipse across a ton of landscape and that kind of thing.
- [07:36]
C
Yeah, in that model we have centralized fulfillment, so we do all of the storage of the blood of other medical supplies. So even in those countries they don't need to think about how do you store that and how do you make sure that you have quality products there. But the difference between flying and someplace like that is you can get somebody life saving supplies in minutes, where a car would take hours. Just in terms of how you would have to drive there.
- [07:57]
B
Yeah, exactly. So I think let's start to look at actually what happens, set the scene more for when a user is actually using the applications that mean delivery comes in. Just to be clear, what we're going to be touching on today, and given that we're software engineering daily, we're not going to be touching quite so much on say like the hardware piece or the exact avionics, this kind of thing, we are going to be talking all about what is the software used by especially the end user and for Zipline internally to make this possible at all. Because there's just so many bits to this that to me is super interesting because the failure modes here are so different I think compared to many other software companies. Especially the outcomes of what could happen if something fails is quite different to just like a server going down. We're talking like a whole plane, autonomous plane can go down. So let's look at the end to end experience. Walk us through, where does this even start? Who is the customer, the user, where does this start and how does it all work?
- [08:58]
C
Yeah, I think if we're focusing on really where we're going and with this platform too that we've built out in the more high precision drone. But the way how this works now we built out a marketplace and so from a customer's point of view, it's super simple. You download the Zipline app, open it, you put in your address, you get to see what partners can deliver to you. So that could be Walmart. If you live near one of those, it could be different. Restaurants that we're serving is going, you figure out what you want to order, you place your order and that is it. So we want the customer experience to be extremely simple and familiar. People know how to do that. From there, we get the order into our system and so we have a set of delivery services that is then ingesting those orders through these Kafkas or QA mechanism, Right? So all these orders come in, we look at it and figure how we translate these orders into missions. Because on our side, what's unique for something like Zipline is we have the fleet of drones, right? So from that fleet we need to figure out what is the ETA that we give back to the customer so they can see, oh, I'm going to get my medicine in 20 minutes. And just calculating that ETA is its own complexity. But as we look at what other missions do we have and orders coming in, what are the drones, battery levels, how far is this going to make sure we know how far to fly. All of those services are core to this whole experience, but it's all invisible to the customer. They don't see any of that from there. If it is a restaurant as an example that we integrate with, we'll send the order directly to their point of sale system. So it's very familiar to the restaurant. Also, one of our goals, the operation side, is to keep it as simple as we can for the restaurants too. So they'll get the order, they'll prep it. Right now we have Zipline employees will go pick up that order, bring it over to our drone, put in the drone, they have an app that they use to be able to tell the drone which order is in there, so the drone knows where to fly and deliver it to. And all of that is fully orchestrated in the backend. We are also now rolling out what we call our zipping points. And you can think of a zipping point as looking like a mailbox. And so the zipping point actually allows our partners to put the orders in themselves. So you don't need to have a Zipline employee come and pick it up, put it in. So they'll go to the zipping point, punch in a pin code, put in the order into that zipping point, close the drawer and walk away. And that's all they're doing again on the backend, that then communicates with the cloud. We know this order came in, we figure out which zip can come pick it up, we send the zip, zip goes, picks up the orders, delivers it to the customer, and that really helps also get the time down and just how fast we can get these orders from being prepped into the customer's hands.
- [11:26]
B
Wow. So a couple of follow ups here. When we do talk about autonomous like and fully autonomous Literally, how true is that in the sense like is this autonomous flight vehicle is everything is software controlled and this thing just gets its mission, I think is the terminology you just used. And from that point to the delivery of the good, and then I guess the return of the vehicle back to its base, I guess that's all purely
- [11:53]
C
software driven, I guess fully software driven and autonomous. So, yeah, there are no humans hiding behind a curtain with a joystick trying to control where the drone is going. That is all the drone. So the drone knows it has its sensors to be able to navigate, to be able to deconflict with other aircraft, to know if there are any health checks that it is doing and any results of those. So in all of that fully autonomous, we have people who are more managing the fleet overall and just ensuring that the overall fleet is healthy in a good state. There's a few different commands they can take, but the 99% of this just the drones know what they need to do and they're able to execute that themselves.
- [12:32]
B
Yeah. Okay, and then you've mentioned, like the P2 platform, and I believe this sort of differs either from the previous platform or just a general image that comes to mind when we think about drones, that there's this launch and catch system. Could you maybe just speak to that?
- [12:48]
C
Yeah. If you look at videos of our Platform One that is really operating internationally right now, you'll see that kind of launch and catch system. And it's really interesting when you watch how the Platform One drone works every flight, actually, it gets disassembled into four different pieces and then reassembled for the next flight. You can swap out a different battery, put in another battery, and you have humans involved in doing that. And that models work super well for our use cases that we have there. And we're still able to do thousands of deliveries a day on that platform. But when we looked at Platform two and what we needed it to be capable of doing, we're not talking about, sure, thousands of deliveries a day, which we're already doing, but we need to now get this to be millions and hundreds of thousands. Right. Like, we want this fleet to be capable of million deliveries a day and more, and to do this all really at scale, extremely quickly and safely and with that high precision. And so what we wanted to be able to do for our customers was just very different than what our current system was doing. And so with that, it meant we needed a different design of the drone. We needed to think about it from the beginning as being a very autonomous, safe system. That can charge itself, that is fully contained in that way and can operate in much more urban areas. It can deliver to that high precision area. So we can deliver to people who have very little space in their yard. And we took a lot of lessons that we learned as we did as company at Platform 1, designed the drone and we've flown millions of miles on there that we could learn from the design decisions there. And there are still essences of that in Platform two. But ultimately the use case that we wanted to be able to do for our customers was just so different.
- [14:24]
B
So I think you were saying there's a launching catch was P1, is that right? And then what is I guess the mechanism change then for P2?
- [14:32]
C
Yeah. So for P2, what we have are we call them our towers and docks. Very simple because it looks like a tower with some docks on there. And so the zip is sitting in the dock and that dock is a charging station. So it'll charge the zip as long as it is sitting there. When the zip takes off, it does a little maneuver away from the dock, goes, picks up the package and delivers it to the customer. And then when it comes back, it just simply goes and redocks and charges. And so our dock structures, we're actually just rolling out now with our 12 dock structures. And our idea with this is we're actually going to really have them be behind the scenes. They are going to be charging infrastructure that people don't really ever see. They're more in the back areas where like there's not foot traffic, you're not around those. And all you know is that the zip just came pick up its package and delivered it to the house. But where they are is actually just charging on these towers, waiting for their missions.
- [15:21]
B
Okay, very cool. Let's talk about like you did touch on it in terms of fleet management. And I think the scale here is something that maybe isn't even understood yet, which is zipline has actually to my understanding, flown 130 million autonomous miles already. I'm sure that number is even higher perhaps than that number I've just pulled. And no injuries and all this kind of stuff. I think it's just trying to understand. So let's just hear what does that fleet number progression look like? And mainly how are we to look at what things break when you scale fleets like this? Because again, when we think about software, I think a lot of listeners today can probably think through what it looked like for them when they were scaling more like a SaaS platform. A very simple zero hardware involved SaaS platform. Oh, we had 100 users, and then when we got to 1,000, then we got to 10,000 and certain things moved around and that was understood. But this feels completely different when you're going through these numbers. So could we step through that?
- [16:21]
C
Yeah, it's a really interesting space, I think, especially when it comes to that hardware in the real world. Tying into the software, software aside, is always such an interesting place to be. And then taking a step back from this, I'll say, like, it feels so similar to when. When I was at SpaceX and we were launching Starlink. One of the first things that my team was doing there was we had our two test satellites that were up in space and we went and sat with the satellite operators and we looked at how do you actually send commands? And it was a very simple web interface that an intern had built at one point, and it worked for that. But then we thought about, where are we going? Right? And I'm going to have thousands of satellites. What do you need to build to actually be able to manage that fleet? And you can imagine that's how that's grown, the systems you need for that. And I think we're at the same place at Zipline now where we have hundreds of drones. But, like, where we're going, we're going to have. We're going to have the biggest fleet of aircraft in the world. And so how do you think about the systems you need there? So we are really designing this firm where you move from a world where you think about drones as snowflakes, right? Each one's a little different. You understand these drones and you have to move into a world where your drones are more like cattle, like you manage them, but each one is not special on their own. That means the systems have to be incredibly smart to understand how to reason about each one and surface what is needed for humans. And so we're building out a network operating center which will really show you the health of them. Any zips that might have any alerts coming up, we have the zips actually telling us when they need to be out of service. So if there's something going on, like the zip will take itself out of service and surface that to us. So that way we can have maintenance go look at it if needed, or we can run health checks on it to reboot it and get it back up. But we want to make sure always safety first. And so that's an important thing with these systems, is designing that so you can really reason about a fleet and not just single drones. I think as we're expanding to more and more metros and we expanding not just nationally but then globally, you're also thinking about how do I now surface this in terms of the level of detail that you need? Because at some level some people will want the detail of specific sites, people want the level of detail on metros, people want the level of details at a state level or country level. And you have to think about like how do you build your systems to be able to show that the most important information kind of top of mind so we can ensure we're giving the best customer experience. We have as much capacity as we can give because we are getting quite a lot of order volume. So that's really important for us and that we can also get zips back up in service as fast as possible.
- [18:56]
A
In mobile application security, good enough is a risk. Guard Square uses advanced multi layered code hardening techniques and automated runtime, application self protection and mobile application security testing combined with real time threat monitoring to deliver the highest level of mobile app security. Discover how Guard Square brings all these together to provide mobile app security for your Android and iOS apps without compromise at www.guardsquare.com every AI team eventually hits the same wall. The models are solid, the infra is solid, but the data coming in is hours old because the pipeline is batch when it should be streaming and nobody's had time to fix it. That's not a modeling problem, that's a pipeline problem. Estuary gives you CDC batch and streaming in one platform. 200 connectors live in hours, not weeks. Your AI is only as good as your pipeline.
- [19:55]
D
Estuary.dev turbopuffer is how companies like Anthropic, Cursor, Notion, Atlassian and Ramp ship their most ambitious search features. TurboPuffer is a serverless vector and full text search engine built on object storage. It's up to 95% cheaper than traditional search databases and just as fast. With TurboPuffer you can index and search 50 million documents at 10 millisecond P90 query latency for less than $100 per month. Head to turbopuffer.com sed to get your first month free.
- [20:29]
B
Do you have any kind of, I don't know, main example that comes to mind? Like something that really did just break when you move from say the five first prototype or they're in production I guess, but the five main precious aircraft. And then you move to this castle analogy. We can't be so precious about these because the volume of flying is more important than I don't Know the quality of the trip. Exactly. Something to that effect. But what broke in that process or something that broke.
- [20:57]
C
I guess, I think for us, the important thing is that we always do still have that high quality, high safety bar. So what really was an example of something that broke was when we had five, 10 drones. You can have people monitoring each one, looking at the telemetry coming out from each drone, looking at the logs after each one, just being like, oh, were these missions okay? Like, how did they go? Oh, this one seems odd. Let's take this one out of service and like go inspect something there. And as the fleet grew, obviously, like, that was not sustainable. And so we definitely saw people getting overwhelmed with just being like, this drone's not doing anything. Does anyone know why? Right. And if you don't have the tools you need, do that. So we built out what we call an auto discrepancy system. And so we were able to hook in to the alarms that we had on the zip software side to say, this alarm is going based on some threshold. So we built a configuration system that you could set this alarm, this threshold. How critical is that? And when that alarm then hits, it fully is integrated into our maintenance system. Then we built that all in house so we had the full control over these systems to then go and create discrepancies. And a discrepancy is something that will then take the zip out of service. You cannot send missions to it anymore. It's like your hard stop of being like, this zip is down. And so once you have that, now we could see firsthand which zips were down. Maintenance then had the immediate signal of saying, oh, it's down for this alarm. And now instead of humans having to go and monitor it, it turned it around to say, well, now the fleet is basically monitoring itself and telling us when it needed to something. And that's just one example of a solution that you need as the scales.
- [22:30]
B
Yeah, that's really cool. And I guess it's software and maybe brackets, firmware that sits on these vehicles when it comes to actually the maintenance, I don't know, cycles. I'm quite into sort of traditional aircraft, if you want to call it that in this context. And obviously the maintenance of aircraft is just such a huge part of what keeps them safe and flying every day. The software piece, I don't really understand maybe the software piece of traditional aircraft. So if we think of the software piece of these aircraft, what kind of maintenance cycles are we talking to manage? Again, thinking at the sort of fleet
- [23:03]
C
level, I think when you look at the maintenance software that we needed for this, obviously there's off the shelf products you could get for maintaining aircraft. It's very different type of use case than what we're trying to do for drones, which are very, you'd say small aircraft compared to what you, you would picture. And since our model is really about something that's much more configured to what we are building and given that we do fully own the entire vertical of it. Right. We know the manufacturing, we know all the pieces on the built materials that went into this. We know any issues that I had, we had the full traceability just made sense to build our own then and to make sure that we were handling all the cases that we needed to. And this maintenance software, then it not only has the discrepancy piece that takes aircraft out of service, it has all the tools that are maintenance technicians need to be able to get them back in a service. So you have work orders that you have planned for, whether it's scheduled maintenance or unscheduled maintenance, where they can see exactly what are the instructions, what do we need to do? Because since we manufacture it, we also provide the instructions to maintenance of like what do you need to do if you see certain problems. And all of this, including the system has gone through review with the FAA to ensure that we are also building for the safety requirements that the FAA needs us to be building for and handling all the regulations that we need to, to be operating in the US for that. And so a system where now not only is it the day to day operational system of the maintenance techs to be able to get SIPs into services, also our traceability system. Because now you know exactly what were the problems, what was the work done, which parts were swapped, how many flights parts have flown on. So we make sure we never overfly parts too because every part has their own life limits on that. And that ties directly into our manufacturing system that we have built ourselves too. So we know what it has gone through even before it gets into commercial. So that's examples of that system.
- [24:55]
B
Yeah. Wow. Moving on to the software that actually drives your operations to make all this work as well. And you did actually I think talk beginning about building an ERP inside SpaceX and I'm aware that within Zipline the same has happened again. And I think this is really interesting, like building an ERP from scratch. This is not something I think that most companies would do. So I'm sure many of our listeners are familiar with the term build versus buy. So especially nowadays most Companies, they need a bunch of software just to run their company, whether they're software themselves or not. And oftentimes there will always be this question, especially at a certain scale of like, are we buying this in or are we going to build it? Or did we maybe build a version of this? And now that's not scaling, we should just go buy off the shelf. Someone who does this bigger than us, for example. So, talking about ERPs, first of all though, I'd like just to make sure we do have a very diverse listener base when it comes to ages and all sorts. So let's just define an ERP to begin with and then let's hear the story around why you thought with Zipline it made sense to actually build one internally.
- [26:08]
C
So an ERP system, enterprise Resource planning system, is the different tools you need to account for your inventory, think through your supply to manufacture. It's all the applications there. So you have procurement, like purchasing applications, warehousing applications, finance applications, like all of these things in one. And the important thing, and I look at build versus buy decisions and you're right, that is pervasive, right? Everybody has the same question, these same problems is what I've seen is when you build your own systems for the areas that are your core competencies as a company, that gives you such a competitive advantage. If you look at SpaceX and Tesla and what they've done about building all those internal systems, how do you manufacturing like, at their core, they're manufacturing companies. And so to be able to build the system there now, you can design the process that you know is the best because you can change the systems to match that process. When you buy systems, you have to change your process to match the systems. You don't have control over those. And there really limits just like how far you can go. And so I don't think you should build every single thing, but you should look at your company and say, well, what are our core competencies that we would really benefit from being able to build custom from? And what are not our core competencies? And let's not build those. There's great products we could buy for those, but these other ones really can enable us to go so much faster. And that's when you look at Zipline, I talked about it being a little bit of this vertically integrated company. I think that is such a huge advantage we have the fact that we do build all of our infrastructure ourselves, we build all of our drones ourselves. That manufacturing is so key to this company. It always has been from platform one to Platform two or with everything we are going, we have great teams who understand how do we really design these drones and infrastructure to really be safe and low cost and to be able to scale with where we're going. And so it made sense to say, well, we should just build the systems as well to ensure that we can build them as quickly, as cost effective, as safely as we can and have all of that information. Because one of the challenges still when you buy ERP systems, I think you'll find that companies tend to not love the part where data is not fully integrated across all. You sometimes will go to one application, you'll see stuff there, you're like, I wish I had this data in this other application, but now I'll have to re enter it over here to make sure it's there too. And which one is my source of truth and how do I. I just wish they were connected. And that's really thing I see from building our own ERP system. You're really building all of this to be integrated fully together. And so that's when say like, hey, we know from manufacturing our full traceability of that. We know any issues that happened, we know our end of flying test results that came out of that. By the time the zips go into maintenance, it's not a separate system, they're all integrated together. So you have all that information. If we were to just buy our own, buy an ERP system and try to build or buy another maintenance system, you're now spending all that effort trying to figure out how do I connect these two when their data models could be very different and not really have a great way of doing that. And usually if you build those kind of integrations, you're spending most of your time just troubleshooting why certain data didn't get to where it needed to be. And when we can own it ourselves, we're able to manage all of that and build it in a way that the data model from beginning to end does make sense. And so I was really excited that Zipline chose to really invest in this space. I think we're already seeing huge dividends pay off from it.
- [29:38]
B
Yeah, I in a past life helped a company. So Oracle, for example, is an ERP provider that I think a lot of companies would maybe look to. And I wasn't super familiar with ERP systems at that stage. I just knew like, well some companies use them and we were providing software more for the consumer end piece. It was just this light bulb moment where they had kept telling me, well, all the data's in the erp, we just need it sync over to the consumer bit. And I looked at the two bits and said, well, you do realise that the way you structured the data in the erp, it's almost impossible that you're going to break out. We're taking line items and you say, okay, well, you've got this parent line item, but you haven't broken it out into the four variants. So if you want to push that through to the end stage, you have to go back and redo your whole ERP basically to make this possible. And they couldn't get their head around this and then the penny dropped with the person on their side. And to me that was just a huge. I was like, wow, you can be a company and buy an ERP and be led maybe through by Oracle experts in that example, and yet still come out with the wrong solution in the end and be left with this. And if you just. I'm not saying maybe building in their case was the best solution either, but they probably would have ended up at least with the right solution if they built it as well. And in this case with Zipline, it's like there must be just so many complexities in terms of data modeling, I imagine.
- [30:57]
C
Yeah, I think you described exactly the problem that people face a lot of the times. And I think ERP companies, they do their best at really building great general solutions for companies. And for some companies, those will work really well. The challenge is when you have the more bespoke company or the more bespoke way that you want to work or you want to manufacture or how your variants are right, that becomes more challenging to figure out, how do we make this general model work for this? And by the time when you get through the implementation stage of it, a lot of times your company may have already changed, like, well, no, it doesn't work exactly how we need it to and then try to figure out, yeah, how do you just mush them together? And so that's not usually the most fun part for engineers to be looking at. I don't think it makes sense for every company at the beginning stages to build these things out. Zipline used a different ERP system for the majority of the time. And it wasn't until, you know, we really wanted to get into the scale manufacturing side of things where he said, hey, now is the time that we should look at building our own. So there are different stages, companies to go through and make the best decisions for that. And it's, it's interesting because I think you Know you describe like, yeah, you didn't know what ERP systems were. I didn't know what ERP systems were before I started working on them. I think they sound boring in that way. And it was never my dream to go work on ERP systems, but what I realized working on them, they are actually so key to how a company can function and work and be efficient at what they are doing. And the power of having good systems is huge for companies. And like my passion has always just been building software that solves problems for people. Just love when people are like really frustrated. I don't love when they're frustrated with something, but I love seeing when somebody is frustrated by a process, a problem solving something. I know we can build software to help with that. We can build applications that'll just make that better. When you see people using that and just all of a sudden now being like, this is so much better, this is so much easier. I now have all access to the data I need. It helps me make decisions faster. And I can see the impact of that. That's so fulfilling. And a lot of times ERP systems are at the core of that.
- [32:59]
B
Absolutely. I've also met someone who works in hospitality and he had moved from a company that did have an ERP to one that didn't. And their reasoning to begin with was cost, which is often an Oracle for example, is six figures potentially. And yeah, just that mental shift of to go from not having an ERP to having to go back to spreadsheets. He just couldn't fathom that. And I think he did manage to convince to get the ERP implemented. But it's just this sort of fundamental data and process layer that so many companies need and just seems crazy that you try and operate without one at a certain scale.
- [33:30]
C
Yeah, I think that is right and I think with, you know, to me I look at it as like it's also not just an ERP system. It is all your custom software that we're talking about building. They all get very tied together in this way. So even, you know, you talked about like what happens when a customer places an order and the systems behind that. But even if you think about the orders coming in, they need to know about the ZiPS to be able to then send the missions and have all these backend services together. So it's still tied to the fleet that we have. And so the fact that from end to end we are building out both back end systems but also front end applications so that way we can, we can manage certain things about our partners or the fleet, or about zips or any of those things. And when you can build all this software in a very standard way with standard tech stack or standard patterns, sudden it just becomes so much easier to continue to build applications for what you need. When you see problems, instead of going to more spreadsheets, you can now think about, well, actually I already have these systems that have three quarters of the data in there. I could just spin up another one. And then how do I think about how that fits into this whole puzzle that we're building? So it is important still to think about the systems that we're building. I think ERP is one part of this, but it really is just this platform that you can then leverage into then saying how do we build custom software for this whole end to end process?
- [34:46]
B
So talking of building custom software, I think one bit I was super interested to think about when I knew I'd be doing this episode was just this idea of most software developers. We understand the concept of like you've got a dev environment, you then push to staging and then try out there and then push to production. And that just seems. I'm just trying to visualize how does this even work in the world of autonomous flying. And I'm going to take a guess there's maybe some simulation that has to be part of that process because loading this onto an actual aircraft for say staging, maybe that works, maybe that doesn't. And that's kind of what I want to hear about. What is that process and how does a maybe fully understanding that? For example, the ERP is the piece of the puzzle and it's fully integrated, but maybe looking at specifically the software that does actually have to run on vehicle, how does that look when it comes to a dev process?
- [35:40]
C
Yeah, so I think there's two major pieces and so I'll touch on one and maybe go in more detail on the other one. Because one of them is more of the software that actually goes directly on the vehicles and that is more of the autonomy, the flight software, embedded software side of the stack. With all of that there, we have a release cycle about every six weeks. We'll cut a release. We'll test that very heavily both in simulation as we're developing on that, but then also at our test sites I mentioned like safety for us is top priority. And so we have a certain set of validation that it has to pass before we will start rolling out a software release into commercial. Similarly, where rice and more on the cloud and application side we have software that is directly integrated with the vehicles. And we have software that is not at all touching the vehicles. And so the validation story there is. It's a little bit nuanced because it depends what you're working on and how mission critical something is. And the way I think about things is if there's a human in the loop, you tend to not need as much verification as if there is no human in the loop. And so that's how you can think of different software. And in that sense, like we have, similar to what you're saying, Rick, you have your dev, you have staging, you push to production. We have those pieces. What we have we call an integration environment. And that is specifically for our test sites. So we can push software there to our test sites first and actually test with hardware in the loop. We've even set up commercial test site, if you will, at our test site. So it operates exactly like commercial. So we can try things internally before again we impact any kind of customers with that. But we are also now at this inflection point that you can call there where with how the fleet is growing and how we're scaling, we now need much more of a simulation on the cloud side too. So we've had a simulation engine and we built one out more fully now for the autonomy and flight software side on the cloud side, we, we're building out a fleet simulator to simulate all the different parts of an order from like order to delivery. So we can actually test that with the scale we are going to. So if Today we're doing 3,000 deliveries a day, but we know by the end of the year we want to be doing let's say 25,000 deliveries a day or 50,000 deliveries a day. I want a system that can test that. I want to simulate that we are doing 50,000 deliveries a day because I need to stay way ahead of that curve and see what services break. And so we're now building out that simulation platform to make sure we are testing all of those services at scale.
- [38:02]
B
That's like effectively load testing. But there are no playbooks for this. Right. In terms of autonomous, again building just, I'll just call it regular software load testing. There's so many platforms out there now that are like you can just plug in any load testing platform. But yeah, I imagine there is almost no buy option on this, right? This has to be another build, right?
- [38:23]
C
Yeah, it's really tricky because it's like you want to simulate your services, which some things can do. But then we also want to simulate our zipping points or humans or what those process are like. And we want to simulate our zip software that we have also so we can play it all together. I think that's where if you buy a product, I think you'll spend almost as much time trying to configure it to make it work than you would have just building something. And when we build software like we try to build it very iteratively, right? So you like build a piece, see if that works, you have one service running in it, then you start building the next pieces of your simulation platform. You get more services in there. And so it is definitely a continued approach to development with it moving on
- [39:00]
B
to I guess the actual human bit of this, which is the team and teams that do build all of this software. How have you thought about structuring those teams? I'd like to get also onto just what does effectively incident response and being on call and all that kind of look like again in this context because it probably looks a little bit different. But especially I guess you'll see how it was all done at SpaceX. And I read a bunch of books around how that maybe looked different there to say a lot of other companies and that's why they were so successful these big mission that they had to go with. And I imagine it's a little bit similar here. You've had to think about building software teams maybe a little bit differently. Yeah, and I just kind of want to want to hear about that.
- [39:38]
C
There's a lot of ways where it looks similar in some ways where it definitely looks different. And I think when I look at how to really build software teams, I think making sure that people can really focus in some domain, have a lot of autonomy and ownership in that space, understand it deeply to figure out what needs to be built is key to having this really high performing teams. And when we're doing this, we make sure the engineers are also responsible for the product decisions too. Because then they can be really close to it and understand with stakeholders, with customers, whoever is using it, like what is the problem? How do we then build a solution that solves this end to end. And so we've structured the team to have three big areas. There's the commerce platform around the marketplace side that we're building. A lot of the, the integrations with our partners, merchants with healthcare partners, gonna be all on that side. You have the delivery network that is more around like how we orchestrate with the fleet. How do we always reason about the state of the fleet, the, the mission processing, like that side that we've discussed is is in there as well as the maintenance side to just make sure you have a healthy delivery network and then you have your enterprise system side which we've talked in depth on. So all of these is to be able to have enough depth within a team to keep it small enough where you understand that, but also really to be able to have quite a bit of ownership of what you're building. I think when you touch on incident response, one of the challenges as we are scaling in some ways incident response can look similar. Sure, yeah, you get paid, you go, you figure out what the problem is, you fix it. Like there's that. But as we're scaling we're now seeing like how are we building our systems to ensure we don't take down for example all of commercial. Right. Like if we are operating everywhere, you now have to think about like how do you shard your databases differently, how do you have a disaster recovery plan, how do you think about operating in multiple regions and how you structure this, how do you release software to have canaries and be able to roll back. Right. So you have all of those problems that in other industries you will have similar problems like that. It's just, I think the tricky thing always in ours is you also now have hardware in the loop as part of that and that always adds an extra layer there that in your traditional SaaS companies or just pure software, you don't have to think about that.
- [41:52]
B
Is there like a sort of skewing in terms of the makeup of the team that has come in so far that have already had some aviation background or does that not really actually apply here?
- [42:02]
C
It doesn't really actually apply. We have software engineers, some of them who have been passionate about drones. That's always a nice connector for people to be like, I want to work in this space. But it's not, not at all a requirement. And that is saying like we are building out this team of software engineers because there's so much we have to build. If you think about the scale of what I'm describing, that we need to build towards and all of these systems that we have a lot of 0 to 1 still to build while we're also building them for the scale of what we are having today and what we plan to handle in the coming months and years. There's so much to do that we are looking for like great full stack software engineers who can really jump in on building out these applications, these backend services to be able to do all of that. And when I talk with software Engineers. What I'm really looking for is that kind of ownership and that kind of curiosity and passion. Just be like, I want to solve hard problems, I want to think about how to solve those. I want to own that end and I want to be really close to customers and have an impact with what I am doing and see that impact. And I think that's where we see the most successful people at Zipline who just thrive with that kind of thing and that kind of ambiguity. And so to me, it doesn't matter what language you've used in the past and which domain that you have worked in. It is just like, do you have that kind of curiosity and passion and hunger of how do I solve these problems and build robust systems for that?
- [43:24]
B
I always hate getting asked this question because it's sort of like why you're asking the question. But I think in this case, the size of the team, I think that's interesting if someone's coming into this industry where it's already a new industry, if they're going to join into autonomous vehicles. But are we talking 100 engineers? Are we talking 20? What does it look for them?
- [43:40]
C
If you look across Zipline, we have a few hundred software engineers, but that is across the autonomy embedded flight software side of things. I'll say the majority of engineers actually sit there within the application or cloud space, be closer to 40, 50 software engineers. It's not a very big team. And that's on purpose. We know we need to grow, but we also are trying to be very thoughtful with this. A lot of the time you can build so much faster when you actually have smaller teams, as long as you are giving them that real ownership and focus and say, that is still something we are working on. At Zipline, we have so much that we want to do, that we want to do everything. So how do we really narrow down to what we just need to be able to execute on, on really well. And so regardless of the whole organization side, we want to make sure our teams are working in even smaller groups than that. Because I think if you think about the times that people have done their best work, in my experience, that has been like when you are on a team of two to four people and you just have this ownership of just getting these things completed, you know, what does success look like? How can you help there that you feel you can move so much faster than if you're on a team of 10, 20, 100 people and you're responsible for just a tiny part of that. And so even as we grow and as we scale, and then we'll always keep that in mind is like, how do we keep that kind of size where you just feel like you can make those decisions?
- [45:00]
B
Yeah, absolutely. Literally, just as we're recording today, something's just shipped where I work and I work it across a few projects. But one specifically, it was pretty much just me and one other person, an engineer and me. And it's so interesting when you have just these two people and you can just ping pong, ping pong. And this thing is going to go out to quite a large scale. But it is so interesting how actually some people might think that was a 20 person project. It was two. And I think, as you say, I think that's where some of the best work is done. So it doesn't particularly matter what the total size of engineering is per se, but how it then gets distilled down into the team sizes I think is super important. So that's good to hear.
- [45:37]
C
Yeah, I definitely picked up a lot of that and learned a lot of that from SpaceX. There was a time that we were building out some of the warehouse management side for Starlink, knowing where that was going to go and scale and went to go talk to this other company about it. And they had a team of 40, 50 people. And they were like, well, how many people is on your team? And we kind of looked at each other. We're like, what's the two of us? I'm like, well, there's like a half engineer who couldn't make this trip, but just us. And that was okay. We were able to build it out still and what we needed. And I think when you do that, you're really forcing yourself to think about what is the highest priority, what is the highest impact thing I can do. Because we don't have all the bandwidth and all the people to do this. So you have to make sure you can ruthlessly prioritize and get down to that MVP and being like, how do I just build the next thing that I know will make a huge impact. And so we. You always have to balance. Like, how do we keep growing to make sure we're not so lean that we can't actually get stuff done, but also that we don't become so bloated that you just end up working on things that are actually not useful.
- [46:38]
B
Yeah, for sure. As we start to land the plane off the episode. So pardon, I guess, completely intended today, but if we look at where is Zipline specifically going and I guess where do you think that drone delivery generally is going because if we sort of loop this back to the start of the episode of where did Zipline started in more rural communities, I think in developing countries and obviously we've now progressed through it's hitting metro areas, in cities and so forth. Where does this go? Like how far does the web of autonomous deliveries, where does it go country wise? I guess just geography wise. Overall, I'm just interested.
- [47:15]
C
Yeah, I think if you look at just generally where the world is going today, there's so much more investments being made into the autonomous space because you're starting to see both with how AI has grown, but also just with what we can accomplish now is huge. And I think people are always looking for ways that they can save time time in whatever they they need. And I think zipline plays directly into that. And so with autonomous delivery, drone delivery, you know, you'll start. You see more companies trying to enter that space. And one of the reasons I'm like obviously very bullish on Zipline is because I think we approach it differently where we really think about this from the customer first side of things. So if you look at our drone delivery like our drones stay 300ft up in the air, they're very high. We want to keep that noise as quiet as possible. We're always looking for ways to make that quieter because we know that's something that's really important to the communities that we do deliver in. And then there's a joy that comes down and just nicely puts your package on the ground. And I think when you see this, it feels magical. Right on. What I love is seeing these videos, especially kids close to my heart, but seeing kids see these deliveries and just be inspired by it and being so excited and seeing that. Parents love it too. Right? Like adults are really thrilled by this. And I think if you look at some of the other players in the space, it is louder or noisier and not as of that customer first type of thinking. And that's something that I'm really glad that we're doing because with where we're going and how we're going to scale, we see this, we have tens of partners right now in our marketplace. We're going quickly right now to hundreds every day. We have new restaurant partners that are on our app. We are going to be expanding into a couple different healthcares this year. More healthcare partners next year, year. So we are quickly going to the place where we are now to in a few years being at a million deliveries a day, which is still a small slice of the instant delivery market, but being able to change the way people think about delivery and how often you could get it or how fast you can get or how easy it can be and how it just fits into your life where you don't even notice, but just what you need shows up. I think it's going to be so big because then not only nationally, but we're going to be doing this internationally too. That to me, Zipline really is at this point where we are revolutionizing an industry. There's only so many times that you get to do that in your life. And I feel really lucky to be part of this and where this scale is really going.
- [49:37]
B
I like when you talk about, for example, how quiet vehicle can be and customer centric and it's also kind of environment centric as well. Because if people see a device like that landing, let's just say in their neighbor's house, but it does give a certain sound or something, you say, oh well, who the hell are this? Company. Company dropping into my neighborhood. I don't want anything to do with them. But if it's something where they go, oh wow, that thing is so quiet. I never even realized it was there until I just looked up. Because I live in high rise in Singapore and it's actually an area where people just fly recreational drones a lot. But the pitch of those drones is so high and quite disturbing actually that I get quite frustrated at these drone flyers, like, oh God, could they not fly them somewhere else? But then by focusing on and the sound piece, that's huge. That's really great to kind of hear that.
- [50:23]
C
Yeah, I think that is one of the front of mind things always in our approach of this is just working with the communities that we're in, hearing feedback from customers or not customers, the neighbors of customers hearing what they are seeing. So we can keep making it better and better. Because when you do something like this, it has to just fit seamlessly into people's lives. If it is more of a nuisance, it's not going to scale and it's not going to win. And we see this as something that really could just enhance people's lives. And so we have to make it that.
- [50:52]
B
Well, Kyle, it's been great to have you on. This is just obviously an area of technology that is only going to increase as we've talked today. But I think just the fact this is probably one of the first times we've really talked about it in any kind of depth on software engineering daily is just great to have done this. And yeah, we're going to be following Zipline, I think very closely. Where is best for people? Look up Zipline or like obviously we got a lot of software engineers listening if they're being inspired and want to like think about applying for jobs, that kind of thing. Where's the best place to go?
- [51:21]
C
You can go to our website, you can look up Zipline. All of our roles are there. Look up software engineering roles. I'd be very excited to have listeners of this podcast come come and contact us. You can also experience Zipline. Right now we're really serving mainly in Dallas and we'll be expanding to other metro soon as we talked about. So you'll see this happening more and more. But if you find yourself there, download the Zipline app and check out our delivery.
- [51:44]
B
Amazing. Thank you so much and yeah, I'm sure we'll catch up again in the future.
- [51:48]
C
Definitely. Thanks for having me.