← Software Engineering Daily
Software Engineering Daily
Anduril with Gokul Subramanian
00:43:05
Anduril is a technology defense company with a focus on drones, computer vision, and other problems related to national security. It is a full-stack company that builds its own hardware and software, which leads to a great many interesting questions ab...
Loading summary
Transcript89 lines
- [00:01]
Shawn Falconer
Andareal is a technology defense company with a focus on drones, computer vision and other problems related to national security. It's a full stack company that builds its own hardware and software, which leads to a great many interesting questions about cloud services, engineering workflows and management. Gokul Subramaniam is senior Vice president of Engineering for software programs at Andareel Industries. He joins the show to share his knowledge of the national security problem set, how Andareal operates and what the company has built. This episode is hosted by Shawn Falconer. Check the show notes for more information on Sean's work and where to find him. Gokul, welcome to the show.
- [00:52]
Gokul Subramaniam
Hey Sean, thanks for having me. Great to be here.
- [00:55]
Shawn Falconer
Yeah, awesome. I actually, I probably should have said welcome back to the show because you were actually here like four years ago.
- [01:00]
Gokul Subramaniam
That's right in the middle of COVID Actually. I recorded it from my apartment. So it's great to be here in a proper studio with the right recording setup.
- [01:07]
Shawn Falconer
Yeah, yeah, for sure. I think people's home equipment setups have really evolved in the last couple of years because so many people working from Rob.
- [01:14]
Gokul Subramaniam
That's exactly right. I remember slowly upgrading my home office over that year.
- [01:18]
Shawn Falconer
Yeah, exactly. Went from like a laptop on your kid's bed to like actual like functioning office.
- [01:24]
Gokul Subramaniam
You start thinking about your background. What are people going to be staring, staring at? I gotta like, you know, put some posters back there. You gotta get the lights, you know, beaming at your face.
- [01:31]
Shawn Falconer
So yeah, yeah, I remember I was at Google at the time at the start of the pandemic and like the senior director of engineering, like his first meetings were legitimately, he was sitting on like a child's bed in a child's bedroom. And I think I'm sure it evolved significantly since then.
- [01:46]
Gokul Subramaniam
Yeah. And for us, you know, we're such an in office culture and now we completely are back to that in office mode that, you know, we were wholly unprepared. And I remember during that Covid period when the rules hadn't been established completely yet, you know, we had people taking tables from the office and driving them to their houses. Cause they didn't have a home office set up and taking their monitor from the office back to their home. It was crazy.
- [02:07]
Shawn Falconer
Yeah, I went through similar things. How was that transition to bring everybody back to the office?
- [02:11]
Gokul Subramaniam
I think it's been good. I mean, we are such a tactile organization in the sense of the things we build. You can feel them, you can touch them. Even as a software organization, we fly our stuff at the range regularly. So I think that it's been much needed for us. You know, we're so collaborative across hardware and software. I will say though, that Covid period really made us invest in remote operation, remote telemetry, remote management, which enables us to now run test sites out of state. Right. So when we were recording during COVID our Test site was 45 minutes from the office. But that's just not possible anymore. You know, our new test site is, we have two. One in Nevada, one in Texas. The one in Texas has one of the largest runways in the United States. The entire United States. And we're going to be flying our airplane next year. So you just can't ship all your engineers out to the test site. And so we've had to really double down on that remote testing capability.
- [03:05]
Shawn Falconer
I would think that is probably forced a lot of organizations to sort of like upgrade, I guess, both like their asynchronous communication and how to think about like remote work. Remote test sites, like even organizations where maybe historically you had to have some sort of field engineer essentially located a customer site to get them set up. You're going to have to change things significantly in terms of not only how you work, but also probably the support systems that need to be in place to deploy software remotely and monitor it.
- [03:36]
Gokul Subramaniam
Essentially 100% true. And yeah, it's kind of a platitude now, but necessity is the mother of all invention. And so when you're basically told and during COVID it wasn't like work stopped entirely, but we had a safety protocol. So it's like one engineer can go to the site and there will be four test engineers and that's it. And they have to be separated by this much. And it couldn't have prepared us better for hitting the real world and scaling to these more realistic, more safety conscious government systems.
- [04:07]
Shawn Falconer
So given that you were on the show four years ago and maybe not everyone listening to this had checked out that episode, I do recommend if you are listening, you should go back and listen. But can you give, you know, before we go too deep into things like can you give a little background on Anduril and like, what exactly does the company do?
- [04:22]
Gokul Subramaniam
Yeah, happy to. So Anduril, we are a defense company. We're a defense technology company. And our thesis is we want to do for defense what SpaceX effectively did for spaceflight. Which is, if you think about the 60s and 70s and 80s, defense was the place to be. I mean, there was a real national need. We were in the thick of the cold war, and all the brightest minds were going into defense. What we have happened over the last 30 years is many of those mines shifted to working in commercial, whether they went to the big bank companies or otherwise. And what Andrew wants to be is a beacon to all of those people to say there is a place for you to come back and work on these really critical national level problems, problems that only a nation state can tackle. And you can do it in a culture that matches the modern world, and you could do it with access to technology, and you can really push the needle. And we hope to lift the boat across the industry. You know, I think that the existing defense primes are feeling this as well, and I wish for them to be able to access that talent as well. And there's a whole set of companies that are coming in behind us. And so we really, I mean, the broader thesis of Anduril is reigniting that arsenal of democracy because we feel that, you know, we're moving into a world of great power competition. And, you know, that's a bunch of jargon, but if you just look at what's happening in Ukraine, you look at what's happening in Israel, if you look at what's happening in the middle east, it's very clear that America needs to continue to hold its role in the world.
- [05:49]
Shawn Falconer
How do you go about, like, appealing to that talent pool? You want to recruit, you know, the best engineer product people, best people, you know, across essentially any functional area to be able to have, you know, sort of, I guess, like fan quality talent pool, but at a non faang company.
- [06:05]
Gokul Subramaniam
Yeah. And actually, you know, it's very interesting because we don't want, quote, unquote, fan quality talent across the whole company. Actually, one of our biggest recruiting draws is the diversity of talent and the diversity of opinions across Anduril. So I'll give you a statistic. 20% of our staff, maybe 25%, are veterans, so they formerly served in the military. You know, I would say another 30% come from FAANG companies. I would say 30% come from traditional defense companies. And I think it's that intersection that really excites people. And it's feeling like you get to work across talent and people that you wouldn't meet otherwise. You get to work across. You know, for a lot of people that I talk to that work in pure SaaS companies, you know, getting to touch hardware is a huge draw for them. And getting to work with the hardware side of the company for the hardware side, getting to work with faang quality engineers is a huge draw for them. So I think that's been the key to our success is that intersectionality of discipline.
- [06:59]
Shawn Falconer
Okay. And then I want to talk about, you know, lattice and lattice SDK and some of the things that you're doing there. So just to start, like, what exactly is lattice? How do you typically, like, describe that?
- [07:10]
Gokul Subramaniam
Yeah, you know, there's a philosophical definition and then there's like a very specific technical definition that we can go into. You know, at the highest level, if I just take a step back and think about history, the defense world has been very good at building hardware. You know, I call these appliances. So, like, if you think about what the defense community has done since the Cold War, they build airplanes, they build submarines, they build, you know, battleships. And they think from a very hardware centric viewpoint. And the net result of that is that these systems are designed in silos and they don't talk to each other in any way. They were never designed to talk to each other. It's nobody's fault. Everyone did great work. But from the beginning, they were never designed to talk to each other. And what it means is that if you think about how a modern conflict will evolve, this is a huge Achilles heel for the United States. And you could see that the military has been talking about this for literally decades. So if you think about what Anduril set out to do is inverting that paradigm. So you build the software as the core from the get go, thinking about every different domain that you'll want to work across and build a common core. And then you design the hardware to take advantage of that software from the beginning. And the net result of that is you get huge economies of scale. So it means that I don't have to reinvent the same code. You get higher core quality, you get a better cost structure, which is really important to our customers. And so fundamentally, that's what Lattice is. It's a common way we think about how we build systems. Whether they're command and control systems, you know, laptops and servers we're putting all around the world, or they're robots. Whether it's our school bus size submarine that's coming online and going through sea trials right now, it's our spacecraft that'll be launching next year. They're all sharing a common DNA and a common code base. And that means they work together. And that means that we get higher quality. And then to get to your point about the release that we're super thrilled to do today, what we realized is that that shouldn't be Locked up to just Anduril. And so we are releasing that Lattice SDK, that capability for anyone to take advantage of. And we have a thoughtful onboarding experience. And we're releasing with a number of partners from the get go who are all taking advantage of Lattice to be able to do those sorts of missions and Lattice itself. So if we get into the very technical definition, you know, breaks into a microservice architecture and I can go into the layers of that architecture, and what we're offering is for people to take that a la carte, so they can take different parts of that architecture, whether it's our networking capability, our command and control capability, our autonomy capability, and they can start integrating that into their systems.
- [09:39]
Shawn Falconer
And if I'm someone who has some of this like essentially legacy hardware, I already have, you know, airplanes and all these other things that military organizations would build up. Can I use Lattice or do I need to essentially be building with Lattice and then building with essentially the hardware instantiation of that from the ground?
- [09:56]
Gokul Subramaniam
Yeah, no, I mean, we do this all day long. We integrate with legacy systems because the reality is, and we knew this from day one, you know, we're super thrilled to see the government adoption of our hardware. But the reality is the vast majority of the kit that the average soldier has access to is not Anduril stuff, at least not yet. And so we need a way to onboard other third party systems. And so the short answer to your question is yes, it's totally possible. And there are layers of integration with Lattice that folks can take advantage of. So if you want to be able to send data into our mesh network so that that data can route seamlessly wherever it needs to go, taking advantage of Starlink and mesh networking technology, you can do that and you can buy a compute module from us, or you could actually use your own compute module and drop our software on there and you can be onboarded into the network and you can go all the way up to, if you build, and this is one of our partners builds, Unmanned surface Vehicle and they want to integrate more deeply into that stack, they can go do that and they can work with us. We can give them recommendations on hardware if there are any changes necessary in order to get that deeper integration.
- [11:05]
Shawn Falconer
What's the actual architecture of Lattice look like and how do I go about like using it to like, you know, essentially start a project with it?
- [11:12]
Gokul Subramaniam
Yeah, that's a great question. So, you know, at the base layer, you could think of it as we have a set of OS level primitives that we recommend you don't have to adopt them, but we have thoroughly tested against them. So Red Hat operating system, for example, or Prometheus for how we do telemetry. So those are a base layer of lattice. Then if you get into the, you know, domain specific layer, you've got three basic components. The first is a networking capability. The second is what we call command and control capability. How do I know where you are and send you basic commands? And then the third layer is the autonomy capability. So then how do I go from sending you commands to letting you move autonomously within the boundaries of a plan or, you know, a set of behaviors? And you kind of think of that as like Maslow's hierarchy. It's like I gotta be able to talk to you, then I gotta be able to send you like basic commands like move here, you know, fire this missile or whatever. And then I got to give you high level intent. And that's kind of the direction we see technology going.
- [12:12]
Shawn Falconer
So in terms of like the commands, am I essentially. Is it like sort of a client server model where I'm sending commands from the server and the client essentially is the piece of hardware that's going to react to the commands and then is the client sending information back as well?
- [12:26]
Gokul Subramaniam
Yeah. So we think about the lattice stack as two different deployment targets. So the first target is what we call the node and the second target is what we call the robot. The thing I want you to take away, the main point here is the node has the human involved. So it is where the human you or I would interface into a set of systems, whether they're manned or unmanned. And once you realize that there's a human involved, it creates a huge set of follow on, knock on technical decisions. For example, you've got to have a user interface because a human cannot talk via APIs. And if you have a user interface, you've got to have a database that's going to store a ton of data so that a human can rewind and go forward and all those sorts of things. So that's the node. The other thing we have is called the robot. And that's a different deployment target, kind of different stack that gets composed. Now the thing that connects that is what we call the mesh. And that's the announcement that we made today. Once you are on our mesh, and the mesh is what's called a pub sub network, it's a publish subscribe network. Anybody can publish topics on that mesh. Anybody can subscribe to topics on that mesh. And so if you are a node which is where the humans operate and interact. They can subscribe to a set of topics. I want to know where the submarine is, I want to know where the airplane is, I want to subscribe to the radar data coming off the airplane. And that data will get seamlessly routed back to you. We do things like, you think about a typical cellular network thing your phone uses. There's a concept of quality of service. So we will ensure the most important data comes to you first and then the second most important data, so on and so forth. And that's how an end user or even a partner gets onboarded into the stack. And that mesh takes care of things like security, it takes care of things like quality of service, et cetera. So that's really the fundamental piece of the lattice stack that then lets you access all the higher order things in.
- [14:09]
Shawn Falconer
Terms of that pub sub, like what are you running underneath the hood? Is that something that you rolled your own or are you using like an existing pub sub provider?
- [14:17]
Gokul Subramaniam
Yeah, this is a core piece of IP for us. We spend a lot of time studying existing providers and we ended up rolling our own. It's one of the few areas we've patented as well. And so it's kind of custom implementation. We do use protocol buffers as the mechanization, as the serialization mechanization, and then we use things like, you know, we work with all sorts of open source authentication systems, right? So you know, if you got PKI or if you've got OAuth or any sort of those, we can integrate with them. But the actual networking layer of the networking protocol, that's all custom built by us.
- [14:50]
Shawn Falconer
And since you're using protocol buffers, are using things like GRPC for communication.
- [14:55]
Gokul Subramaniam
Yeah, precisely, yeah. So you can send data over protocol buffers over the mesh, you know, you publish data, but then you can also call commands. Right. So like if you want to task something, you want, you know, a direct ability to expose what I can task and then be able to leverage that task and get an act back and that we would use something like Google a grpc.
- [15:17]
Shawn Falconer
Okay. So you have essentially this like data flowing back and forth between these nodes and the server. Where does this actually run? Are you running these within your own cloud deployments? How does that sort of stuff work?
- [15:30]
Gokul Subramaniam
Right. So one of the fundamental premises that we had to make and design for is that we cannot assume availability of any cloud stack. And so the entire system is designed to be agnostic to both any individual cloud vendor, if we have them, we'd love to use them and also the ability to run on premises. The node itself can go all the way down to the form factor of a laptop and all the way up to the form factor of a data center, an on prem data center or in the cloud. The robots tend to use Nvidia form factor, the Nvidia Jetson family of chips. But we're compatible with any sort of embedded form factor that we can run on. And you know, that's why I call out that distinction because you've got very different compute available to you depending on if you're a robot versus a node.
- [16:19]
Shawn Falconer
Are your software deployments secure by design? Lately, secure by design and shifting left principles have been hot topics in the software industry, pushing development teams to make security a foundational part of software development. Today's sponsor, Bitwarden supports developers in securing every phase of the development lifecycle with end to end encrypted credential management. This ensures software is built on secure principles to prevent data leaks and unauthorized access. Try Bitwarden Secrets Manager, built specifically for developers to safeguard infrastructure and machine secrets, or Bitwarden Password Manager for everyday logins and other sensitive information. Start a free trial today@bitwarden.com and within this kind of like mesh infrastructure where this information is flowing between all these different nodes, like, is there essentially a component of like AI based decision making that's also part of it for supporting things like defense operations?
- [17:10]
Gokul Subramaniam
Yeah, and this is an area that we're all kind of leaning into. The DoD is leaning into this as well. But we're all trying to figure out where do we draw the line and how do you do this in a safety conscious way? You know, so there's this concept that is basically traditional systems have been human in the loop, so the human is directly inside the decision making loop. The idea with AI and a lot of the latest generative AI technologies is how do we move the human from being a blocker inside of the loop to becoming sitting on the loop? So they are observing the loop entirely. They can stop at any time, but they are not directly an actor within the loop that they have to do something. So the closest example I would describe is if you've got a Tesla, for example, you could put your car in self driving mode. You are on the loop. The car is going to make a set of decisions, but you need to be watching it extremely closely and you should be ready to take over at any time. And so as it comes to AI, that's the way we're starting to think about it. And we're Starting to lean into a lot of new use cases. Your viewers may have seen our announcement with OpenAI, our partnership there, and we also have partnership with Palantir, and so we're really excited to lean into that. And then the second thing I could say on AI is, you know what I really think about this is my personal opinion. If you think about what's happened in the last five to 10 years since the advent of Transformers, I don't know that the innovation has really been on the model architecture side. And there's certainly been new model architectures and things like that. But the real breakthrough has been on the data side. And it's been moving from increasing the volume of data that's been available to train these models multiple orders of magnitude, hundreds, if not thousands x more than we had before. And how did we do that? We got the labelers out of the loop. So what we basically did is we created a situation where the data could be self labeled. To give you an example, Right. I'm going to look at a Wikipedia article, I'm going to take the first paragraph, feed it to the model, ask it to predict the second paragraph. I don't need a human labeler in the loop because I can just check, did you guess the second paragraph correctly? And by doing that, you can explode the amount of data you have available to train. That's our goal with we are sitting on top of, we hope, a hugely valuable data set for defense purposes that no one else has access to. Only the US military has access to this data. And we want to really treat this data with respect with the right authorities. But start to use this data to start being able to generate the next generation of AI models for our use cases.
- [19:32]
Shawn Falconer
Yeah, I think the primary bottlenecks today with both on the model side, evolution of the models, and then also with building AI applications are really like data challenges.
- [19:43]
Gokul Subramaniam
That's right.
- [19:44]
Shawn Falconer
Essentially like if you're hitting the limits of what you can scrape publicly, then you need to go find data somewhere else. And then, you know, there's challenges around generating purely synthetic data for training in terms of degradation of, you know, model performance and integrity. And then even on the other side of essentially building AI applications during prompt assembly, we now have the advantage of like really large context windows where.
- [20:07]
Gokul Subramaniam
That's right.
- [20:07]
Shawn Falconer
But then it's like, how do you provide the right context, you know, with the right information in real time to generate the most reliable response. And both of those things are really around essentially data problems.
- [20:18]
Gokul Subramaniam
Yep. Everything has followed from how do we get access to High quality, huge volumes of data. And then you figure out like, okay, well, I need to train against these huge volumes of data, so I need these huge server centers. I need to figure out how to network the server centers together. And I need to be, you know, there's this concept of coherence when you're training, but all of that is second order that follows from can you get access to these huge volumes of automatically labeled data effectively? So the big idea that we're after is, is how do we do that for our use cases, this national security problem set where we think we are closest to those problems. And as we talked about with our mesh networking architecture, we're just constantly sucking up that sort of data.
- [20:57]
Shawn Falconer
And even outside of AI, what are some of the engineering challenges you face with trying to scale lattice to handle the complexity of these dynamic environments where you got classified servers, battlefields, you know, you probably know all the terms better than me, but I mean, these are like not sort of the typical places that most engineering organizations are deploying things.
- [21:20]
Gokul Subramaniam
Yeah, I'm so glad you asked this question. I think this is what gets people fired up when they come work here and they see the scale of our problems very differently than what you'll see in big tech. So I think one of the fundamental building blocks for us is we build for a decentralized world so we have no concept of all the data. There's not going to be one server to rule them all, where all the data will come back. You know, if I go back to my Tesla analogy, every Tesla car can communicate back to Tesla headquarters and they can kind of pull all that data together. We live in a world where our systems have to be designed to work against degradation of communication and reforming these meshes, wherever they are, maybe they can connect back to Washington D.C. chances are they can't in the real world. And they'll have to work in a dynamically evolving, kind of decentralized paradigm. And what that means is the way you design applications has to be designed with that in mind. Let me give you an example, right? If someone sends you a text message on your phone and your phone is off, you have an expectation that when you turn your phone back on, that message will be received and you'll be able to see it. Many DoD systems are not architected with that in mind. You cannot make that fundamental assumption. Furthermore, you can't make the assumption that that DOD system can ever see Washington D.C. it may be able to see another DOD system if, you know, the bad guys aren't jamming it. And so how do you architect and build applications for that world? That kind of becomes our seminal challenge. And if you think about the Lattice SDK that we're releasing, it contains those lessons learned that we've spent seven years learning, those lessons about how to build for the decentralized world. And we're trying to share those lessons. And it's a different paradigm fundamentally.
- [23:00]
Shawn Falconer
How do you test for those types of environments?
- [23:03]
Gokul Subramaniam
Yeah, so I mean, this is where the real world test range comes, is incredibly critical. We have a really robust simulation architecture that we're continuing to lean into. And one of these ideas is that we can't have one central team do it all. So we have people building the architecture. So if you're interested in that kind of work, you come here and work on that. But then we actually have people embedding close to these problems, really getting in the weeds with the end users, often traveling to the end users, working with them when they do their testing. Remember, I mean, we've got to build our stuff so that we're not there when it's actually being used. So we train with our end users and learning from them what those edge cases are. And then we bake it back into our integration tests and our simulation environments.
- [23:46]
Shawn Falconer
What's the simulation environment? Is that something that you had to build yourself in order to essentially like simulate the types of situations that like the end user is essentially going to be in?
- [23:55]
Gokul Subramaniam
Yeah, and I would say, you know, we've got multiple layers of this. We have a system called the software integration environment. It's a virtual machine based thing where we can spin up, you know, we talk about these complex scenarios, battleships in the water, airplanes in the sky, nodes on the ground. We can actually spin up VMs to replicate that. We can replicate the networking links, we can degrade those networks on demand to put ourselves through those paces. And then what we've built is really a game engine that can start to describe these scenarios. And we can model at varying levels of fidelity these systems and how they'll behave. And then what's really cool is we can put into that game engine the real code, the real autonomy code, the real C2 code that would run live and we can put it through its paces such that you can build confidence. And that's actually really how we train our operators, is they work through the system in sim, but they have the confidence that it's running the real code and so the real vehicle will behave no differently.
- [24:49]
Shawn Falconer
How reliable is sort of this like simulation environment in terms of like, if Something goes wrong there, are you pretty sure that it would be something that would actually go wrong in sort of the real world?
- [24:59]
Gokul Subramaniam
Yeah. The way I would describe it is if something goes wrong in the sim, it will definitely go wrong in the real world. But if something goes wrong in the real world, it's not guaranteed you're going to see it in the sim. And it's an enormously long tail. You know, I've heard Elon describe this as if you could perfectly simulate the real world. It may be proof that we ourselves live in a simulation. So it may not be actually possible to simulate the real world entirely. And we don't try and chase that long tail. What we do is we capture the 80, 20 rule, we capture the 80% in the SIM, and then we know when we go to real world testing, we're not wasting our time learning those silly lessons. And then we're actually getting into the details of what we can only find.
- [25:40]
Shawn Falconer
Out in given, like, the sense of nature of defense data. Like, what are some of the unique security challenges that you run into in terms of, like, having to safeguard this data as it's, you know, flowing through the mesh?
- [25:52]
Gokul Subramaniam
Yeah, I mean, I think the DoD has done a really good job of thinking through. I mean, this is. There's historical context, decades of it, of, like, classification levels, how to run a secure network. You know, that's NSA's job to, you know, write that policy. Fundamentally, I think, for us, the challenges then become how do you operate in that world? So let me give you an example. I've got systems that are in the Middle east that are defending U.S. bases today. Those systems are seeing data from adversaries that are trying to do nefarious things. How do I retrain the model for those systems based on learnings that I'm getting in the field? Do I have to push my model training architecture all the way out to the US Base in the Middle east where that data is being collected? I sure hope not, because then I've got to get Nvidia GPUs and a whole server set up out into the Middle East. That's probably not feasible. Can I get it back to Android Headquarters in a way that every Android engineer can see? Absolutely not. That is not permissive, and that would be not what your customer would allow. So there's a middle ground. We've got to find somewhere. A lot of this gets into our secret sauce of how we operate and how we work, but that we've got to figure out. Okay, we've got to do training fundamentally at the edge. Not everyone's going to be able to see all the data. That infrastructure has to be portable. That's why we don't depend on any cloud provider. Those sorts of things become a lot of the challenges we have to work through.
- [27:10]
Shawn Falconer
When you're doing something like training at the edge, are you, does that end up getting fed back to like a central location?
- [27:17]
Gokul Subramaniam
If it can be right. And you know, we've brokered interagency agreements where data collected from one part of the DoD or one agency can be shared with another, but those have to be brokered point to point, case by case. And then we can do that sharing. But in other cases the data may never ever come back. And that's a good thing. Right? We don't, you know, that's the nature of the work we do.
- [27:39]
Shawn Falconer
A lot of limitations. You got to sort of try to navigate.
- [27:42]
Gokul Subramaniam
Yes. So then, and then the next question is like, all right, well I've got to upgrade to the model training architecture. How do I push it? How do I do fleet management? Fleet management is a huge thing we think about and it's going to be an even bigger concept for us as we increasingly proliferate with more and more animal systems. So how do I know the last time I saw a robot what was the version of Lattice it was running or the errors through? How do I proactively remediate those issues and how do I push forward the latest upgrades? We've got this concept where we can stage the upgrade forward. We can stage it at a node and then if a robot connects back with that node, we can say, hey, we've got an upgrade for you, I'm going to send it on over. So there's a lot of technology we've built to do those sorts of things.
- [28:22]
Shawn Falconer
How often do you have to push upgrades?
- [28:24]
Gokul Subramaniam
We're pushing upgrades all the time. And again it comes down to, you know, our rate limiters is working closely with our customer and their comfort level. So if we have a system that's in R and D, then we'll be pushing upgrades very quickly because both us and the customer understand that we're in R and D mode. If we have a system that's operational. These are life preserving systems. We go through an extremely rigorous set of tests. Often it's a real world test at a test range, you know, it's called an operational test. It's very rigorous for when that system is, that code is allowed to go forward. And these are some of like the real material differences between you know, commercial world and DoD that we have to build for. So, you know, we have a concept of named releases, right. Just like, you know, Linux does, where we will support that release for a long period of time. Time. We're making that commitment to a customer, their issues found. We patch that issue, we go back and patch the issue in the release and then we'll forward deploy that, that bug fix with the customer's concurrence.
- [29:20]
Shawn Falconer
Do you have to also take into consideration, like, if something's operational, the sort of like the time it might take to like upgrade and like, if there's like a reboot, restart process that could impact essentially the person who's using that thing?
- [29:34]
Gokul Subramaniam
Yeah, exactly. So, you know, if I think about one of our sites, we'll have to coordinate with them when the right time to do it is, and so they'll work that into their shift. Essentially it's a build base and then we have to basically have operator training. Right. So if we're sending out new updates, that's materially changed the way the system is used. You know, we're printing out sheets, we're giving it to the users, we're having trainers go out there to help them get familiar with the new changes. You know, we did a massive overhaul of our command and control interface. And this is one of the cool things about working with Android for our customers is because they don't have to pay for this. They'll get this for free. Free. But you know, what we had to do for one of our systems is we had to run both the old system, the old C2 system, and the new user interface side by side. And the user had to be able to use both until they were comfortable switching over to the new one.
- [30:22]
Shawn Falconer
Yeah, I mean, I get frustrated enough when like I have to restart my slack and zoom, whatever, all the buttons have moved. Yeah, yeah. But I load something out like my life depends on, essentially.
- [30:33]
Gokul Subramaniam
Yeah, exactly.
- [30:34]
Shawn Falconer
So you're now, you know, releasing the Lattice SDK, I guess. Like, what was the motivation behind watching that and what are some of the things that I could do with it.
- [30:42]
Gokul Subramaniam
And the motivation for launching that? I mean, I mean, there's two fundamental problems we're trying to solve. The first one we mentioned at the top of the podcast, which is that traditionally DoD systems are not built with interoperability in mind. They are built in silos. They're built as, you know, I'm building this system and then I'll think about how to connect it later. And the solution to that is not everyone has to coordinate at all times. Like, that's a crazy thing. Like, the DOD is building thousands of systems at any given time. The idea that we're all going to, like, talk constantly, not going to work. You know, our belief is, the real answer is, let's put forward a reference, let's purport a specification. And we basically say, if you adopt this specification, you get compatibility for free. And we will keep the specification alive, we'll keep it modern, will incorporate all the feedback. And that basically decentralizes this problem of how do we get compatibility. So now that we have the ability to be compatible, how do we move data between each other? And that's where the mesh comes in. And that's the second part of what we're releasing. So, you know, to answer your question about what you can do with it, if you are a builder of a robot, you can get compatibility with everyone else in the ecosystem and ultimately make your robot more useful. And that's what we've seen with a lot of the launch partners that we're, you know, coming out with. They can make their system more useful, their system more applicable to problems that our customer wants them to solve by adopting this technology. And what they get with that is compatibility and the way to send and networking, the way to move data between each other. And that's what they'll be able to do.
- [32:09]
Shawn Falconer
Yeah. So you basically solving like the interop problem rather than around like a, sort of like a standardization of how you're going to do interop.
- [32:17]
Gokul Subramaniam
That's right, yeah. And, you know, we really looked at and learned from how did the web world deal with this? Right? Like, how did the browsers become interoperable? How did the TCP IP stack, the USB stack? How were all those things birthed? Right, Like Plug and Play, you know, I went back and watched the old Plug and Play videos from Microsoft where, you know, they announced that people like, their minds were blown when the printer plugged in, and all of a sudden it just worked. And you're like, I can't believe that's what the 90s were like.
- [32:39]
Shawn Falconer
Yeah, I'm old enough to remember those days.
- [32:42]
Gokul Subramaniam
Do you remember it? Blue screen? Actually, I went back and watched that video the first time they plugged in in a blue screen. Windows.
- [32:47]
Shawn Falconer
Yeah, the blue screen of death. Amazing in terms of being able to bring this, like, lattice SDK as something that, you know, someone external to Anduril can use. Like, what were some of the challenges with sort of like essentially packaging that up? It's different to have something that you're using and you built internally, everybody knows how to use and you're deploying against versus like, here's this thing that someone else that we never talked to is going to now, you know, roll with.
- [33:10]
Gokul Subramaniam
That's right. And I think we're still on that journey. Truth in advertising, Sean. Like, we are still early in that journey. But the first part is realizing no privileged access, so you can't backdoor and get access to your favorite engineer on the other team to answer your questions. You know, we've got to like start working through the Doc site and if the docs don't have the answer, we've got to like, you know, submit that as an issue and fix the Doc site. So, I mean, I think there's a famous Jeff Bezos Memo from like 20 years ago where Amazon had to go through this to build fundamentally what turned out to be aws. I think we are going through that journey right now. You know, we're lucky in that we're launching with a set of launch partners that we have trusted relationships with. And as we onboard new partners into the ecosystem, we're going to have, you know, second party basically relationships with. But our goal is to get this to the point where anybody can take it off the shelf and start using it.
- [34:02]
Shawn Falconer
That's the Lattice Partner program.
- [34:04]
Gokul Subramaniam
That's the Lattice Partner program, exactly.
- [34:06]
Shawn Falconer
And sort of the main organizational benefit, there is feedback for how you continue to evolve this product.
- [34:12]
Gokul Subramaniam
Yeah, so the partners get a bunch of different things. One is first class support and kind of a privileged relationship with our team. The second is access to our simulation infrastructure. So we'll be spinning up stacks for them. They'll be able to use our SIM to integrate their systems in and be able to go back and forth with us. The third is basically kind of business development, a privileged business development relationship with us.
- [34:34]
Shawn Falconer
Is this primarily targeting other companies that are working on defense products? Or could this also be other companies that happen to be working on other different types of products?
- [34:43]
Gokul Subramaniam
We're a defense technology company at the end of the day, so our mission is defense. But I'll tell you, one of the other hats I wear, Sean, is I run the space business at Anduril as well. And one of the things I'm super excited about, I call it the team of super friends that we're putting together in space, is there are a ton of companies that are commercial space companies. And actually two of our launch partners, Apex Space and Impulse Space, are commercial space companies that they might not have the expertise in the defense market that we do. But by coming into our partner program, they're benefiting from our expertise and they are integrating with our system to make their commercial offerings more readily available to the dod. And I think that's a huge benefit to a ton of companies that are defense adjacent, dual use, have commercial use case and a defense use case, maybe offer a component, but not the whole system coming into the ecosystem makes them more available to the defense market.
- [35:33]
Shawn Falconer
Yeah, I mean we were talking about some of the like, engineering challenges of like remote deployment and these defense scenarios. I mean they get even more complicated when you're talking about remote deployment to something that's interstellar.
- [35:43]
Gokul Subramaniam
Yeah, luckily, I mean the space community has been dealing with things interstellar for a long time. But like, how does that then plug back into the DoD's Link 16 network, for example? Right. Like that's not a thing that a commercial space company is going to want to concern itself with. And that's something like you come into our mesh, we could take care of that for you.
- [36:01]
Shawn Falconer
Can you talk a little bit about the Anduril's involvement in like Desert Guardian and the Valiant Shield military exercises? What was your role there?
- [36:10]
Gokul Subramaniam
Yeah, I'll take them one at a time. So Desert Guardian was a tremendous opportunity for us where we had a number of vendors that all produce distinct systems. So think radars, cameras, radios that could detect electromagnetic signatures. The socom, our customer brought them all to a site and said we want all this to work together. And the way we're going to get this is instead of issuing two year long contracts where we specify the requirements up front and no one really knows what's going to happen and you ask for something, two years down the line you get something different. You guys are all just going to basically run a hackathon and you're going to use the lattice mesh and the lattice SDK and be able to generate interoperability. And that's what Desert Guardian was all about. We learned a ton. I wouldn't say everything worked perfectly, but everyone came out with the result intended, which was systems started working together faster than anyone ever thought possible. And I think that was kind of the first kind of set of proof points to us that like, hey, this is really valuable what we're doing and what we've done. You know, every Anduril system, when it comes off the line, works together. And it's not like I'm like centrally coordinating with all these disparate teams. Right. They just know to adopt the software and then you get the, you know, the interoperability, that's what we started creating now with third parties who we'd never met before. Valiant Shield, very different story. So the DOD runs huge exercises where they are training themselves. And really the purpose of these exercises is to go through the paces and like, train for these hugely complex scenarios. Right. What would an operation in the Pacific Theater look like that you can't do through sim? And it's literally working people through their paces. Valiant Shield, our participation in that, we acted as an integration interoperability layer to enable systems to communicate with each other, to enable that exercise to take place.
- [37:59]
Shawn Falconer
In terms of, like, your engineering teams. And when people are brought into the company, like, and say they come from like a commercial world, like, what is one of the, like, sort of biggest surprises that they run into when trying to adapt, you know, what they know from the commercial world and engineering practices there to working in this, you know, sort of defense centric world?
- [38:18]
Gokul Subramaniam
Yeah, great question. Let me give you a few different answers. So the first one is, if you think about the commercial world, you've got a set of economic relationships that don't exist in the defense world. First of all, there's like a profit seeking relationship where, you know, typically a commercial company has something that's generating a lot of money. I call it like something that's generating a geyser of cash. And you're trying to. Ideally, ideally, yeah. And you're just trying to pump that geyser harder and harder and harder. That's not how the defense world works. So a lot of those, like, motions that they're used to, very different from us. Right. Our customer is, you know, we don't have a profit seeking, you know, our customer has the budget of a nation behind them and they're really concerned with like, you know, are we safe? Are we credibly deterring war? Are, you know, they have a whole different set of considerations. Right? And it's to be noted like, the US Government is a monopsony, right? It's the only buyer. And so you've got a very different economic relationship with your customer that takes people a little bit of time to get around. You know, you have things like, because of safety is such a high concern, right? Like, you can't just transplant like an app store concept from the commercial world and be like, yeah, let's just run an app store in DoD like, we don't want that. Our customer doesn't want that. Right. You can't have someone like ship an app that causes a weapon system to become no longer operational. It's just not okay. So there's just a level of rigor and a mindset that's very, very different. Then the other thing is that I think people love about coming here is just the sheer variety of use cases that we're working on. I mean, I can't think of anywhere you could go right now where you can one day be working on a submarine that's doing sea trials in Australia and then turn around and talk to an air spacecraft that launched and then turn back around and talk about, you know, sentry towers that are defending U.S. bases. It's just the sheer variety of domains, sheer variety of technologies that we get to work on is very different than, you know, a company that kind of just like does one thing because of.
- [40:15]
Shawn Falconer
These, you know, added constraints, does a lot of like off the shelf, you know, tools of the trade not work. Like you talked about, you know, rolling your own pub sub. Like you tried existing, you know, pub sub systems out there, but for whatever reason, maybe they didn't meet your specific requirements. Is that the case with a lot of off the shelf products?
- [40:33]
Gokul Subramaniam
The way I disguise this internally is we decompose our stack into what's called a domain specific layer and an infrastructure layer. And what I say internally is anything in the infrastructure layer. We better have a darn good reason why we're going to roll our own. So, like how we're going to think about storage and databases. You better give me a really good reason why you need to roll your own database, how we're going to think about deployment, infrastructure, how we're going to think about telemetry, right? Like those are all things you'll find at any bank company. Those are things we're going to use off the shelf. And we're going to make a real clear buy bill, trade off. The things that are in the domain specifically are how we think about AI, how we think about networking, how we think about command and control, how we think about autonomy. We are often going to roll our own because it requires us to have a deep partnership and understanding of our mission that is not easy to see replicated in the commercial world.
- [41:24]
Shawn Falconer
So what's next for the Lattice SDK effort?
- [41:27]
Gokul Subramaniam
I think for us, the key is making sure that we are keeping this evergreen. We are responding to, you know, want to do product LED growth, right? You want to be responding to customer feedback. And we're going to start to open up more, more and more of the Lattice ecosystem. So I mean, what we've opened up is a very tiny fraction of the Lattice ecosystem because we wanted to ensure quality against that and we wanted to ensure that, you know, it was kind of a hit, people adopted it. If we see that happening, you're going to see us open up more and more. For example, we did not open up any of our autonomy capability, but if we see adoption, we're going to start to be more willing to open up more of the stack.
- [42:06]
Shawn Falconer
Do you think that's going to be a trend that other companies sort of in the government and defense space take? I think Palantir has over the last couple years also kind of taken a similar path where they're opening up a lot because presumably you learn a lot of really specific things working within the sector that are generalizable to other companies use cases and domains.
- [42:28]
Gokul Subramaniam
Yeah, I think you're seeing us in paltier especially lead the way in this regard. One is sunshine is the best dissent disinfectant. You're going to learn a humongous amount just getting people to look at this. The second is, you know, I talk to our customers about this a lot. We are not trying to create vendor lock. The reason we have a core common stack is because it's critical to our cost structure, it's critical to generating interoperability, it's critical to all these good things that we and our customer want. But we're not trying to lock people in. And I think vendor lock has been a huge pain point for our customers. And even when they look in the commercial world they're like, yeah, Apple has this awesome app store but you know, they basically locked everyone into it and they're like rent seeking on top of it. And there's like court cases going on about this. This is. The DoD cannot afford to have this happen. And so by open sourcing this stuff, we're showing good faith to say like that is not our objective here. And so I think you are going to see a lot more of this in defense.
- [43:21]
Shawn Falconer
Well, Gokul, thanks so much for coming back on the show.
- [43:24]
Gokul Subramaniam
John. I had a terrific time. Thanks for having me.
- [43:26]
Shawn Falconer
Awesome. Well.