← Software Engineering Daily
Software Engineering Daily
Boston Dynamics with Matt Malchano
00:49:56
Boston Dynamics is a robotics company known for creating advanced robots with highly dynamic movement and agility, designed to navigate complex environments. Their robots, such as the quadruped Spot and the humanoid Atlas,
Loading summary
Transcript73 lines
- [00:00]
Matthew Melchano
Boston Dynamics is a robotics company known for creating advanced robots with highly dynamic movement and agility designed to navigate complex environments. Their robots, such as the Quadruped Spot and the Humanoid Atlas, have applications in industries ranging from logistics to public safety. They also garner widespread attention with their impressive videos showcasing robots performing complex tasks with precision. Matthew Melchano is Boston Dynamics Vice President of software. For more than 20 years, Matt has been a technical contributor and leader on robotics projects such as Spot, Big Dog, LS3 and Sandfli. He has led efforts in areas including software, product and robotics, autonomy, perception and control. Matt joins the podcast with Sean Falconer to talk about his wide ranging work at Boston Dynamics. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.
- [01:09]
Sean Falconer
Matt, welcome to the show.
- [01:11]
Matthew Melchano
Hey Sean, thanks very much. I'm happy to be here.
- [01:13]
Sean Falconer
Yeah, thanks for being here. I'm pretty excited about this. We don't have that many shows that touch on robotics. A couple. But I feel like a lot of times robotics in some senses, like a lot of people who end up studying engineering or even becoming software engineers, a lot of times that's like where they start, is some interest in robotics. So I'm kind of curious, where did your fascination with robotics begin?
- [01:34]
Matthew Melchano
Well, when I was young, actually, like essentially like in elementary school, I just became very excited about robots. And I remember telling my parents, you know, oh, I'm going to grow up and be either a detective or a robotics person. And I happened to luck out that I went the standard computer science track all throughout my education. And then I decided when I was going into my master's that I wanted to shift gears and learn about an applied area, right? Like learn how to apply software to robotics. And I shifted gears, did work in that area, and then graduated and was lucky enough to get a job at Boston Dynamics, right when we started building robots at the beginning of sort of the Big Dog era and got to live out that childhood dream of mine.
- [02:18]
Sean Falconer
That's interesting. So how do you think, like, building and thinking about building, like, software for robotics, something that is going to exist essentially in like a physical capacity and maybe even interact with people, you know, physically. How's it different than maybe developing software that we think about maybe is only living in sort of like a digital sense?
- [02:36]
Matthew Melchano
Yeah, it's interesting. So in many ways it's the same, right? We build all the same kinds of software that I think people who work all across the software industry do. So we're making web pages and continuous integration systems. We're building up servers and clients, we're building APIs. All that's a part of building product robots. One of the things that separates, I think robotics from just general software is it's like you run on a computer that might not be there some of the time. And so you spend a lot of time dealing with the availability of hardware, the costs of scaling that hardware. A robot is a thing that can break, particularly when it's in its research era. And so you'll discover like you'll get a chance to run your algorithm and then you'll go, oh, whoops, the robot's gone for the next day. I've got to go and go back to the drawing board to work on that. And so you spend a lot of time trying to figure out ways to be very efficient about how you, you run software on the robot to maximize what you learn every time you do an experiment. And a lot of the work is an experiment in a way that, you know, like all software works an experiment. But software is blessed by being very repeatable, typically. Right. That if you build up a system where you control all the inputs to the system, then you know, you should get the same answer out every time you use it.
- [03:46]
Sean Falconer
Right.
- [03:46]
Matthew Melchano
It's deterministic. Robotics isn't necessarily that way. Right. You know, you go run on the robot and the second time you have slightly different noise on all the sensors, slightly different behavior out of the robot. You know, maybe someone pushes it and all these things, things are things that the system has to respond to but you aren't guaranteed to see every time. And so it's got this extra level of non determinism in it, it's got this extra level of logistical complication. But at the end of the day, like a lot of the software feels very similar. Right. There are some robotic specific software areas that people work in. You know, you're going to go off and learn about doing real time software if you work in robotics, or you're going to work on firmware if you're working close to the hardware. But all of those are also sort of very natural, I think for a lot of software engineers.
- [04:26]
Sean Falconer
Yeah, absolutely. I mean, it's interesting, like you're in terms of testing. Even though Boston Dynamics probably has access to more robots than most companies, you still have a limited number of resources. So presumably everyone's sort of competing for access to the same limited set of robots to be able to test something once they get beyond simulation phase to being able to test it actually on physical Hardware?
- [04:49]
Matthew Melchano
Yeah, absolutely. So it depends on what level of maturity your project's in. The spot project, for example, runs like 100 robots internally. I'd have has a tremendous amount of QA hours that get done every week on those robots. That gives us the ability to be really sure that that system is going to work when we ship a release. Right. And it gives us a lot of coverage just day to day. So you get super fast feedback about whether the changes you've made are making the robot better or worse in the environment that we're running it in. But in the early days of that project, we only had a few robots that we were able to use to build the product robot out of. And then you have to sit there and work very carefully to figure out who's going to run what. And is the software bug buggy or is it the hardware that's broken? Lots of very difficult questions like that.
- [05:34]
Sean Falconer
Yeah, absolutely. You work for, as we mentioned, Boston Dynamics, I think, probably easily one of the most famous, if not the most famous, advanced robotics company in the world. And I'm sure you get this kind of question all the time, but why build robots that can actually walk? Why make humanoid robots, robots that can do backflips? What is the reason for doing these super dynamic behaviors with a robot?
- [05:59]
Matthew Melchano
So one of the sort of obvious things is that robots that walk just have a lot of advantages as a product over robots that don't walk. Right. They're able to step on things, step over things. They're able to move through thin gaps, Right. They can climb stairs, they can reposition themselves in all sorts of ways. They're able to control where their body is oriented, which is really important for manipulation or for doing inspection with the robot. All key uses that we've used on Spotify. Also true for humanoid robots, right? They're able to reconfigure themselves and do things that humans do, like reach down or like access low areas. And so much of our environment that we built both indoors and outdoors is focused on, you know, being able to do things like that. Right. Not all of it, obviously. And you know, many people are differently able, but at the end of the day, like it really is an advantage for a product to be able to walk around. Right. And we've discovered that it's let us gain access to, you know, markets that previously people have not been able to succeed with wheeled robots in. Right. And I think that's been really interesting. You know, the other thing that's interesting about legged locomotion is nature just Provides immense set of examples of the potential of what you can do with legs, right? From speed to mountain goats climbing mountains, to just, you know, the kinds of, like, brachiating motion that, you know, monkeys do. Like, you never think to yourself, wow, I've solved this problem completely, right? Because there's always some angle that's of that form. And so I think those are two sort of key advantages that are just very practical for us to focus on. You know, beyond that, I think that the holy grail of robotics is general purpose robotics. Building machines that are able to cohabitate with human beings and do really meaningful work for people to improve and enrich their lives. And so, you know, I think that, like, us continuing to focus on building really capable robots, which I think that limbed robots have, for example, are just really worth the effort.
- [07:47]
Sean Falconer
Yeah, I think it's an interesting point because a lot, especially internal environments, physical environments, are designed by humans. And we design a lot of our environments based on how we're going to access them. And that's us walking around these different places. And even in nature, a lot of, at least the large mammals that are taking up space are walking, and that's their form of essentially locomotion. So it makes sense that a lot of the earth in the world is basically going to be set up for success when you can move in that capacity. Like, what is kind of hard, though, about designing a robot that can walk like naturally, that mimics essentially the walking behavior of either an animal or a person?
- [08:29]
Matthew Melchano
You know, in some sense, we like to think that we've, like, largely solved mobility at Boston Dynamics. Right. We think that our walking behaviors are working really well right now. And if you look kind of at our focuses, we now spend just as much time on questions like manipulation, autonomy, navigation, and how to, you know, use AI on our platforms. So in some sense, we don't think of mobility, like legged mobility as that hard. Right. However, we put tons of effort into getting here, and there's a lot of nuances in how you build a system that can do that. And there's always more to do. A great example for us was we have customers that have slippery facilities, facilities with wet floors that are concrete. Those are incredibly hard to walk on. One of the things that we did was use reinforcement, learning to build new capabilities into spots. So it was able to walk on those slippery floors much better than it could before and probably better than I could walk on those floors right now. And that's, you know, like us adding more capabilities into what we have. But you know, we feel really good about the performance of the legged mobility that our platforms have right now. So it's been less of a focus for us and we're looking now much more forward towards, you know, how to use that mobility to perform work with manipulation or to, you know, let our robots go and do a set of inspections across a building, things of that form.
- [09:43]
Sean Falconer
And then is there a difference in terms of solving that problem? For bipedal locomotion like a human basically versus an animal like Spots, obviously modeled after a dog. Are there different challenges with that kind of motion and is one easier than the other?
- [10:00]
Matthew Melchano
Yeah, it's an interesting question. There are and there aren't. So the nature of the motion is different in that you have four legs or two legs. But the same questions apply. How do you balance the system with the legs that are touching the ground? How you move the legs that aren't touching the ground to smart places to put them? What are smart places to put them? How do you perceive the ground around the robot so that you're able to, you know, know where you can and can't step? All of those are very similar problems. And if you go look at the legged locomotion literature, you know, way back, right, people generalize and say, oh, animals that walk on four legs are actually sort of walking on one virtual leg. And so there's a way that they're all like kind of equivalent. That said, you have more legs to manage with a four legged platform, but you also have, you know, more points of contact so it's easier to sor of stand in balance. You know, we use very similar software across all of our platforms. So I would say 80% of our software stacks are the same across Boston Dynamics robots. But we do solve the locomotion problems differently. And particularly when you start getting into questions of like doing backflips, for example, you know, those end up being problems that require us to use a bunch of forward thinking that extends kind of further and has a different precision, the kind of forward thinking that we use inside of the quadruped platform. So when I say forward thinking here, I mean, you know, modeling and predicting where the robot will go and the results of actions that it takes. And so there are differences, but lots and lots of it's very similar. And a person can move pretty easily between working on those platforms without needing to retrain, for example.
- [11:32]
Sean Falconer
Yeah, so you said, you know, 80%, I believe is of the software is sort of common across the different robots. So when you sort of crack the code on One type of skill, how transferable is that knowledge? Like once you nail down sort of walking, for example, does that help you figure out how to do, you know, say, like pick something up and throw it or some other type of movement like that?
- [11:54]
Matthew Melchano
Yeah, that's a great question. And the answer is it does. So, you know, there's two ways that it helps you really, right? One way is robotics is a pyramid, right? You know, you're building things on a lower level of basics and then you're adding more complexity and building up. And sort of at the top of this pyramid is, you know, the behavior, right? Something like a high level behavior. But at the bottom is just tons and tons of software to compute kinematics, to make motor control happen, to make sure that you hit your real time deadlines, to read sensors, to integrate imus, just all this stuff. And you have to get all of that correct in order to have the thing at the top work. And so the more you get that pyramid correct, the better off you are the next time you go and build the next behavior. Because you're like, oh, if I reuse all this software I've written and it's all accurate and works correctly, then you're in a great place to move on to building the next, like, behavior. Additionally, some of the ways that we build behavior are, you know, in some ways you just can write robot behavior and be like, oh, to do this, move the arm here, move the arm there. But in other ways, you can write systems that solve problems around behavior. And those can be learned systems or they can be optimizers. And once you build one of those, you sort of transformed your problem from needing to hand code what your robot does every time to saying, oh, I can have this robot solve this problem for me. And so that's where we get a lot of benefit, is building some sort of solution framework and then reusing that over and over again for a set of related problems.
- [13:23]
Sean Falconer
How much of the software that's running is based on sort of more of these general learning systems versus essentially learning how to solve a problem through test and iteration. Kind of like a baby learns how to walk and move around and stuff like that versus something that's sort of like a hard coded sequence that the robot's going to execute?
- [13:44]
Matthew Melchano
Yeah, that's a great question. Reinforcement learning, which is the thing that you're describing there, right? This idea that you use some sort of mechanism to automatically develop what's called a policy, and then that policy is a thing that you can apply to the robot is A huge part of robotics these days. We're doing a ton of it internally, depending on our platforms. We do a mix of things. One of the challenges there is you really just shift a lot of the problems about how you specify the behavior or how you validate that the behavior works in different, different circumstances and you still have to do all that work. And so we continue to use a mix. We're on Spock, for example. We have a lot of sort of classical controls that we put into place, which we now augment using reinforcement learning. In the space of how that classical controls works on some of our other platforms, we're starting to look at reinforcement learning as a way to do various tasks that are relevant to manipulation. But we also mix that with other techniques as well. You know, I think it's still hard to build a reinforcement learned robotics behavior that essentially has everything in it that you need, need to productize that behavior and use it robustly, you know, with a customer who might not be a roboticist and just wants to rely on the idea that the robot will work and do what it's supposed to all the time. But, you know, huge research area, we're doing a ton of it. And I know it's really big in industry right now.
- [14:59]
Sean Falconer
What about the difference between sort of trying to design a robot essentially to be like a general problem solver versus solve more specific tasks? Like if you could design a robot that I don't know is specifically can go into a car that's on fire, rip the door off and save somebody, versus a robot that can be thrown into any sort of dangerous scenario where you wouldn't send a person to try to rescue. And those seem like completely different design scenarios. And you would develop the software and the hardware maybe in a different way.
- [15:29]
Matthew Melchano
Yeah, they really are. You know, I guess the great example of this is something like robot arms in a factory versus putting, you know, a humanoid robot in a factory. Right. And I think there's really a continuum here, right. On the very specific robots, they're very successful, Right. Robot arms work really well in like manufacturing right now. However, you pay all these additional costs, like you have to go and retool your factory whenever things change. Right. Or you got to like go and move all these arms around. And so they're good for fixed activities, for example, that you know ahead of time, exactly what you're going to do. But if that changes and requirements always change, you know, that's sort of a software engineering thing. Right. Then, you know, all of a sudden you're going to realize your fixed purpose robot maybe can't be adapted easily to the new thing. On the other end of the spectrum is building a general purpose robot, which of course is a huge area to figure out, you know, to really achieve the kind of generality that humans have for like working on tasks. But I think in the middle, there's lots of places where we can build robots that are really reproducibly general purpose. Right. And then apply them to specific tasks, achieve ROI on those tasks, return on investment, and then use that to continue growing robotics as a business area. Right. And proving it out in more and more sort of applications. Right. As you get towards general robotics. And I think that's the tact we've really taken. You know, something like our stretch robot is designed, you know, and sold to take boxes that arrive on trucks and essentially like unload them onto a conveyor. However, the overall system is a general purpose robot, and so we think it can be used for lots of other handling, like activities that need to be done. Similarly, spot's really a general purpose quadruped platform, but we spend a lot of time trying to make it as adapted as possible for a very specific task, which is enterprise asset management, you know, going around and inspecting things inside of a facility and making that easy for someone to do. Right. And so I think that, like, there is a continuum and we're sort of figuring out a way to build robots that do a whole range of tasks as an approach to get to general purpose robotics. I think it's going to be like that for a long time.
- [17:30]
Sean Falconer
How many different types of robots does Boston Dynamics have? And like, what are they specifically design for?
- [17:36]
Matthew Melchano
Yeah, that's a great question. We have three main product lines right now. So we have the SPOT robot, which is a general purpose quadruped, and it is primarily designed to do industrial asset inspection. So, you know, you take it and you buy it and it comes with a dock and essentially you put it in your facility and you can wake it up and it'll drive around on a schedule and take photos of things and use a variety of sensors that are payloads in order to understand whether things in your factory are breaking down, whether you have something like an air leak or an overheating motor, things of that form. Right. Gets used in a lot of industrial areas. It also has the ability to do lots of general purpose work, so you can buy other kinds of payloads and people use it to patrol areas, they use it to work in dangerous environments, things of that form. We also sell a robot called Stretch. Right. So Stretch essentially got sold. Starting last year we started selling it commercially and it is specifically designed to operate in warehouses. Right. And it's interesting because it's not a legged robot. It's a pallet sized base with a large robot arm and it's able to use perception and autonomy mechanisms to understand boxes in the environment around it and then pick up those boxes using a vacuum gripper and unload them. The initial application is unloading trucks, which is really important for logistics. There's so many of these trucks that get unloaded. But we imagine that it's going to be useful in all sorts of places in a warehouse. Right. And then our third product line is the Electric Atlas, which we just launched, sort of our initial like public views of in some videos. And it's intended for real world applications and we're planning to be testing it along with Hyundai, the company that owns Boston Dynamics in majority, for a variety of applications that are relevant to them.
- [19:17]
Sean Falconer
And what is the interaction with the people involved? Like, you know, I have, you know, Spot running in my factory. Like how do I interact with it and kind of give it the tasks that it needs to perform?
- [19:28]
Matthew Melchano
Yeah, that's a great question. Our primary mechanism is a product called Orbit, which we also sell. And it's a web based experience. So you log into it and you can use it for fleet management to understand what your robots are doing, monitor where they're moving around on a map, essentially schedule missions for those robots. And say at 4am this robot wakes up and patrols its kilometer route and then comes back and sits down and then it presents that data up to you and manages notifications. And so that's, we think, the dominant mode of interacting with our robots. We're also working on adding a bunch of features there that, you know, allow you to do more with the inspection data as it comes in. And then, you know, out of the box, you can actually just take a spot robot immediately and start driving it. It ships with a tablet controller that allows you to drive it around. Similarly, Stretch is planned to be added to Orbit also contains its own tablet controller to allow you to move it around. It has a simplified interface that makes it easy to train individuals who work in logistics facilities on how to, you know, command the robot to do things like start an unload job or pause that unload job. Additionally, all of our robots are planned to have APIs. So we have an API on spot that allows people to build sort of an arbitrary amount of software there. And many of the behaviors that we've Built also operate using that same API. So for example, our navigation stack. And that, you know, is just an indicator that, hey, this API is quite powerful and lets you do a variety of things. We recently just shipped it with an RL capability so that people are able to take our robots and build their own RL based controllers and then directly control the joints of the robot, which was a big ask from a lot of researchers. And so there's a bunch of different ways, but predominantly, you know, we're focused on finding ways to deliver robots out to not just people who are robot experts, but to people who can just use them to make their jobs easier.
- [21:17]
Sean Falconer
Yeah. So with that API, can I essentially programmatically build my own like extensions and new behaviors that can be executed by the robot? Kind of like a much more sophisticated version of like Lego Mindstorms or something like that?
- [21:31]
Matthew Melchano
Yep, absolutely. And it operates at a bunch of different levels. You know, the API is runs over Google grpc, which is a remote procedure call framework. Right. That communicates over TLS and you know, HTTPs, which is great for being able to be compatible with enterprise environments. And then you're able to command a whole variety of different. We've broken the sort of surface area of the robot's functional API into a bunch of really microservice style services that you can talk to which allow you to take either high level or low level command. If your robot ships with an arm, there's an API that allows you to control how the arm works. You can drive the robot like the tablets all implement against the API. So you can tell the robot to go touch, to go to some location, turn on and off all the features. It's really very full featured and does a nice job of putting the complexity of commanding a robot behind a really usable network interface.
- [22:26]
Sean Falconer
Yeah, that's really cool. And then what is the actual development process for building one of these robots? I'm sure they're years in the making, but where's all this stuff kind of begin?
- [22:36]
Matthew Melchano
Yeah, that's a great question. Oftentimes it begins with the previous robots. We're hugely iterative. We're definitely a build it, break it, fix it company. Our CEO says that to us all the time. And so our machines are constantly evolving. That's one thing to keep in mind. It's not like we throw it out and start over again. We just take what we have and figure out how to keep moving it forward, both locally and then in terms of project to project. We often try to reuse as much software as we can between iterations so that you can get started as soon as you can. You might go ahead and get started and then decide you need to change a bunch of that software. But that means that you're not spending a lot of dead time just writing things from scratch when you start a new robot. The other advantage of starting from a previous generation of robots is that you have a lot of information that can go into your design process. So, you know, Spot's a great example. I essentially led the software product effort on spot to turn it from a research prototype into the shipping product robot. And in order to do that, we started with just lab Spot, right? This thing that was only designed to work in a lab. It was, you know, totally hacked together in order to demo really cool behaviors, right? And we said, oh, there are all these things that are important to building a product. And we just began to incrementally push those into the system and transformed the entire system into the thing that we ship that has security and the API and the ability for a person to configure it and, you know, all these different, like, things that weren't part of the original system. But, man, it was great to start with an actual robot and be able to validate every change as you went. When we decided to design a new piece of hardware, we often start with things like simulation, right? Where we go and build a physical simulator of it and start asking, how would we move it around? We work with canonical behavior requirements so people will go and. And do a study of how the robot should move. Very similar to product definition in other places in software, right? You know, how should I move? What are the important factors here? And we'll start doing a bunch of hardware analysis. You know, like, hey, can we make this hardware work the way we expect? Can we test these sensors out on a cart? All that kind of prototyping, all this involves a lot of software. You know, that's one of the really fun things about robotics is even when you go work with a bunch of hardware, you end up having to work with a bunch of software. And so as a software engineer, you get exposed to this world of. Of, like, electrons and atoms, right? All these, like, machines and mechanisms, and you learn a whole bunch of stuff as part of that. That's just, like, really, really fulfilling, I think, to see a world outside of just like, the software world that, you know, all of us have been a part of. So we'll start with that. And then you just do a huge amount of work, right? You know, like, lots of iteration, lots of prototypes. You'll build similar like looks like works like robots. You'll be bringing things up all the time and making them like work and then working down the long tail of issues until the system really works as reliably as you want it to.
- [25:16]
We're going to take a quick break to talk about Bit Warden, our sponsor of today's episode. Are you among the 65% of developers who still Hard code secrets and source code Storing machine and infrastructure secrets in code, unencrypted env files or messaging apps can make your business more vulnerable to leaked secrets and data breaches. Bitwarden Secrets Manager offers a super straightforward solution to this problem. It prevents secret leaks by making it easy to manage and deploy machine and infrastructure secrets all from one secure location. Bitwarden is unique because it's open source, end to end encrypted, and can be easily deployed into your existing environments with a robust CLI SDKs and out of the box integrations like Kubernetes, GitHub and Ansible. Start a free trial today at bitwarden.com.
- [25:56]
Sean Falconer
Secrets in this sort of culture of like, break it, fix it, like, I think it makes a ton of sense. Like if you're going to figure out how to, you know, have a robot do some sort of new behavior, realistically that robot's kind of break a lot, probably through getting to a place where it's actually able to execute that behavior reliably. So you have to be able to fix it, otherwise your learning cycles would be so slow. But how much of the hardware is being built in house versus built by somebody else? Or is it just even if it's built by someone else, the assumption is that you should be able to fix some of it in house so that you can have essentially these fast learning cycles.
- [26:34]
Matthew Melchano
A lot of our hardware is built in house, so certainly a lot of our prototype hardware gets built here. We have a machine shop of expert machinists. They're sort of F1 grade, like machinists are able to build all sorts of really cool parts. And we've moved all of our manufacturing into Boston Dynamics. So we have a factory that produces spot and stretch robots right now, which means that, you know, they come off a line in a box and we're able to get the same version that would ship to a customer. And once you hit that point, that's just great because now you're never like, you know, you can just get another robot if you need one. And we do all the sort of hardware maintenance in house, you know, it's important to do a lot of that kind of thing because similar hardware and software, the entire concept behind a build it, break it, fix it kind of cycle is to do root cause corrective action, right? You want to take anything that goes wrong and ask what was the root cause of that thing? Like, really drive it back to whatever is the thing that caused it to happen. And then you want to take some positive action to fix that root cause. And so that's just, you know, consistent across our entire organization from software to hardware. Everyone's asking, oh, when something has gone wrong, great, where do we file the ticket? Which team needs to go and figure out how to fix it? How do we know that the fixes have all gone in? And then we, similar to lots of software orgs, structure our software release process around that. We structure our hardware maintenance system processes around that and use just all that to constantly drive the robots to higher and higher mean times before failure and mean times before intervention, which are the kinds of metrics that we're tracking all the time to make sure that the products are always improving.
- [28:02]
Sean Falconer
And in the project where you took spot from essentially being like an R and D product to actually being a product that you can sell to somebody who's not part of Boston Dynamics or part of some sort of government agency, what were some of the big changes and technical challenges with making that actually a product?
- [28:22]
Matthew Melchano
Yeah, that's a great question. We had to do everything from building the manufacturing flows for as you created components, how you'd assemble those components in a factory, all the UX that people need to be people who work on manufacturing. We had to take the robot from essentially being a bare Unix style system to something that had security features so it has an encrypted disk and locks out the ability to access it. The API is authenticated. We had to add the API. People went off and built the tablet UX that didn't exist ahead of time. There were tons of internal interfaces that had to be either removed or pivoted to be shippable later on. There was a long tail of making sure that the system was really reliable. You know, you'd have these early robots and the early robots would have some issue. And you know, if it happens once every two weeks, you know, a lab person won't care, they'll just reboot the machine. But that's totally unacceptable for a product. And so, you know, all of those get run down and solutions get made for them. There's lots of things that are on the edge between software and Hardware. So, you know, people constantly are trying to understand, like, you know, what, like, why is the hardware acting the way it does? How do we, how do we make that work? We had to go write new motor controllers for it. You know, one of the interesting things was spot transformed from being a hydraulic robot into an electric one. So as part of that, lots of work was done to build sort of the electric motor version of the system. So there was a lot of like, sort of fundamental science type work there that was interesting.
- [29:55]
Sean Falconer
Why was that switch made from hydraulic to electrical?
- [29:58]
Matthew Melchano
There's lots of reasons. You know, electrical things have improved since the early days of Boston NAMs. We had lots of hydraulics work. Hydraulic is great for power density. It's also great because you can use gasoline as a fuel, which gives it really high energy density. So, you know, you can put a limited amount of weight in the robot and the robot can run for many, many hours. You know, we saw this in our LS3 project where we had a robot that was able to go, I want to say like 20 kilometers and, you know, on a single sort of tank of gas. But no one wants a gasoline powered robot in their, in their houses or their factories. And so, you know, it really wasn't the right kind of robot for indoor applications. And so we really started making a transformation around 2014 to electrical robots. And in fact, you know, our last hydraulic robot was the Atlas series of robots. And that has now turned into the E Atlas robot that you've seen in videos, which is all electric and is even more capable than the hydraulic one, which I think is like just an amazing testament to how good you can make electrical robots these days in terms.
- [30:57]
Sean Falconer
Of going from a robot that maybe is used for doing a demo or even used in a lab situation where you can reboot it if some sort of problem happens or people are there to service it. To suddenly having potentially a robot that is in manufacturing in a factory running 24 7. How do you deal with things like dust accumulation or other things that could get into the robot's components, essentially impact its reliability and actually cause it to malfunction?
- [31:28]
Matthew Melchano
Yeah, I mean, you know, mechanical engineers spend a lot of time on that sort of thing. Right. Ceiling is a big thing, as is like figuring out ways to. You know, it's always challenging when something moves, things break around, things that move. I think people who work on their cars know that and things like that. And so one of the interesting things is like, you know, everywhere where you have wires that cross moving points or connectors all of those are things that people are constantly figuring out how to drive out of of the design in order to make the robot just as reliable as possible. And so it's very similar to the software process. People are just constantly asking, how do I make the equivalent of git commits to my manufactured design to say, like, oh, let's fix this. They call them ecos in the hardware world, but they go, essentially make commits to the process and go, we're going to go and swap this component out. And all these things are done just to drive up that overall reliability, reliability of the system.
- [32:21]
Sean Falconer
What kind of safety controls have to be built into the robot to make sure that it doesn't, you know, accidentally run into somebody or it's performing some certain maneuver that applying force to some object and that ends up like applying force to a person or, you know, some other thing that could lead to, you know, hurting someone.
- [32:37]
Matthew Melchano
So we have an entire safety department inside the org right now. They work on questions about hazard and risk analysis. Right. Which is a common practice for how you take consumer product and make that product safe for people. And they do those analyses on our different product lines as we ship them. And so that's all like a key piece of how we build what's called an operating domain. Right? But the idea that you combine both the way the machine itself works along with ways that you require it be used in order to ensure that people don't get injured. Right? And so that's the thing we do with all of our products on the stretch product, for example, it's a very large robot. It has a variety of safety features that ensure that, you know, when someone's driving it, they are right there with the robot. There's a switch on the pendant they use that ensures that, you know, they're paying attention to the fact that they're driving the robot. And the robot can only move at a very limited speed. And then it has to go into a like, limited area where it's able to then do its box unload at full speed. But the rule is no human can be in that space spot because it's a smaller robot, doesn't have those same concerns. But there are like, lots of ways that we work together with customers to make sure the robot's safe long term. There's open problems in safety and it's one of the places that we are continuing to look. Right. If we want to really build robots one day that work together with humans in close proximity, then, you know, there's lots of technologies that are relevant to Go and explore in that space. And we're starting to look at things from functional safety up to AI capabilities that'll make sure that the robots are able to like work with human beings.
- [34:13]
Sean Falconer
In terms of AI. Some parts I would assume we mentioned reinforcement learning, so learning new behaviors, but some other parts of the robot's behaviors are probably down to some sort of machine learning algorithm like executing. So how do you balance between the need for sort of being able to react and be real time versus essentially the raw number crunching and computation cycles that's going to take to execute some of these models?
- [34:39]
Matthew Melchano
The easy answer is lots of optimization, right? I guess structurally one of the things we do is we break up the decision making into different times. Right. This is very common for real time software. Right. You might have a very fast control loop that is executing some very like basic algorithm to just, you know, run basic control on a set of joints. And then you'll build slower loops around that that are able to run at different speeds. And so it's very common for our like sort of low level controllers to run at hundreds to thousands of hertz, but then from there to have something like an image processing algorithm running at tens of hertz. Right. And then as you go into sort of more modern ML models, many of those can't execute like you know, more than maybe once every like half a hertz or, you know, 2 hertz or something. And so you kind of take what you need to be able to do and you break it up into pieces and you have those pieces all complete over the schedules and timelines that they can. Right. And of course the thing you're always trying to do is figure out, well, is there a way I could do this faster? Could I speed up this loop? Could I make this decision a little bit faster? You get lots of advantages by making decisions faster in robotic systems. It improves your force control, it makes the robot respond to things more quickly. But on the other hand, that means you just have to have much more efficient code for that. That's the general balance of how things work. And we'll even like access things off robot and that can take longer. And so it's really a systems problem. Right? You go and design the robot to achieve the set of goals that you need.
- [36:13]
Sean Falconer
There how many like sensors are on one of these robots that you're taking as input at any given moment.
- [36:19]
Matthew Melchano
So there are various kinds of sensing on the robot. So for example, there's what we might call low level sensing that would be for every joint on the robot. We care about the position of that joint, we care about the force that that joint's exerting. Oftentimes we may measure those things in multiple ways, right? And then building out from there, you have physical sensors around, you know, the orientation of the body, like an imu, right? And then you'll often have a set of perception sensors built around that. So the perception sensors on spot, there are five pairs of stereo cameras that look out into the environment around the robot and allow it to perceive the ground obstacles, things around on the robot on stretch, there's a mast, right? And that mast contains a set of cameras and other sort of distance measuring equipment that's able to image things around the robot. And so it varies from robot to robot. But you know, you'll have a set of sensors for just how the robot moves. Proprioceptive I guess would be another word for it. And then a set of sensors that are often perception based. And many of our robots, you know, a spot, you can put payloads on the back so you can go and add a lidar, right? And use a velodyne to understand, which is a spinning lidar combination in the AV industry to understand, you know, the space around the robot. Or you can get one of the inspection payloads that we sell and put really high resolution cameras on the system so that you can get really good images of far away things and you know, read them as the robot walks by. And so there's a bunch of different configuration in that space that gives some sense for the kinds of sensing the systems have.
- [37:49]
Sean Falconer
What like programming language are people actually writing the software in that's actually executing on the robots?
- [37:55]
Matthew Melchano
Yeah, that's a great question as well. And it's a lot of languages. You know, one of the things that happened as we moved into product is we became very polyglot, right? And so, you know, classically lots of robotics gets done in C. The vast majority of people who do sort of behavioral type work on our robots are working in C. There's a growing number that also work in Python, right? Or some Python C hybrid. We've also used Python and historically matlab for doing offline analysis. So, you know, you need to go compute some downstream numbers and make some graphs. So that kind of thing. Over time TypeScript and JavaScript have grown as you end up having to put websites and web type experiences in lots of places. That includes both in, you know, on robot and in the cloud and you know, our like other appliance type technologies. And then additionally we're starting to see uptick of languages like Go and Rust being used in small areas where people are like, hey, this is the right solution for these problems. And then all of that gets pulled together into the set of information that we release onto the robot as a software release. And so you're really like pulling all these different languages together into one thing.
- [39:02]
Sean Falconer
How big is the software engineering department at Boston Dynamics and how is it sort of organized?
- [39:07]
Matthew Melchano
Yeah, Boston Dynamics is around 250 software engineers, give or take, depending on how you want to categorize them. It's organized into product thrust teams. So, you know, we'll have a software engineering team that is focused on the spot project and the stretch project. We also have a central software team which I run, which is a team that works between those teams doing things from engineering enablement up to R and D style activities, you know, future technology development, basically just anything we can do to like make the company move faster. Our software engineers vary in background from people who are very robotics focused and work on behavior or controls over to people who are very customer focused and work on things like building software solutions for customers and doing field service for customers, and then a wide variety of experts in all sorts of different domains. So people who build websites, people who write firmware, people who write safety sensitive software, all of those things are things that are part of the group of people who are building software at Boston Dynamics.
- [40:11]
Sean Falconer
In terms of third party software that are using to make some of this stuff happen, is there any, you know, open source technologies or software frameworks that play a key role in this?
- [40:20]
Matthew Melchano
Yeah, absolutely. We use a lot of open source software. You know, I would guess we probably have hundreds of open source packages that are a part of the software we build. We use a lot of, you know, traditional technologies that people who work in computing probably know about, like for example, protobuf from Google, GRPC from Google. We use Bazel as a build system. You know, we got some of that because we used to be owned by Google. And so, you know, that was probably like a flavor of the things we do. We also use a lot of open source and robotics packages like OpenCV, the Point Cloud Library, PCL, eigen, the Linear algebra system. So, you know, we pull a lot of that kind of technology in as well. And that's really useful for, you know, people being able to rely on really high quality software and tools to build things. You know, we don't, we tend to be shy about committing to frameworks in our experience. So, you know, one of the things we don't do for Example is use a lot of ros, but which is the robot operating system, a common framework. And that's in part because once you sort of commit to a framework, you often find that, you know, it's both does a lot for you, but also controls a lot about what gets done. And so we're just always cautious about, you know, how much you commit to a thing like that and then how much you can align your roadmap developing software to, like, where that framework chooses to go. It won't necessarily go where you want it to as a product company. So that's something that we, like, pay some attention to. You know, we also built a lot of software because we've been doing this for over 20 years now, building robot software. And so it's one of those things where, you know, we built a lot of technology and so we tend to use our own frameworks and our own systems as well.
- [41:56]
Sean Falconer
I think being at it for 20 years, just like Google had to, you know, in the early days of the web, you have to invent a lot of stuff because no one's solved some of these problems before.
- [42:04]
Matthew Melchano
One of the joys has been actually seeing software that I might have written like, 15 years ago. I've been at the company for 21 years, you know, seeing software that I might have written a long time ago still in use today. Right. It's an amazing feeling to be like, wow, you know, I wrote that on some field test out in the Mojave desert, you know, 15 years ago, and it's still in production on a robot today. Right.
- [42:25]
Sean Falconer
Some.
- [42:25]
Matthew Melchano
Some utility or something like that, which is true. And so that's been just a very interesting thing about the sort of long duration of time that we've, like, worked on these projects.
- [42:34]
Sean Falconer
How much do you have to worry about, like, backwards compatibility and stuff like that as you're building new versions of the software? Is that a big factor? Like, do the robots that are already out in the wild get remote updates?
- [42:47]
Matthew Melchano
We design all of our robots to accept remote updates. We have a system we invested in that lets us do those updates with high reliability and completely control the set of software that's on the robot. It's also a key part of our security story. So there is a backwards compatibility to the robot hardware question that always has to be maintained. So, you know, we, we don't want to ship a release that won't work on some generation of the robots. That's generally been not a problem. Since the robots are, you know, we. They evolve over time, but we've been able to factor those evolutions into the same software package. Right. That's a concern within our software environment. Backwards compatibility, you know, we build everything from tip of master, which is a very common sort of model, largely in an integrated sort of, mostly in a single repo, which means that we can do compatibility of our own software with itself relatively straightforwardly. And so mostly the concern then has to be around, oh, did we release an API in the past and is that API still valid? We have policies and procedures around similar to probably everyone around, taking function calls in that API, deprecating them, and now announcing it to the customer, et cetera. And so we manage it everywhere we need to, and wherever we can, we simplify it by just making it not a concern. Right. Because that kind of stuff, like, rapidly becomes incredibly complex. If you let the matrix of compatibility, as we like to call it, turn into a big matrix, all of a sudden, you can spend a lot of your engineering time just working on validating that matrix.
- [44:13]
Sean Falconer
Yeah, absolutely. And then in terms of the simulation software, where presumably a lot of the testing is happening, how well does that work versus the real world? Like, if something works in the simulation, what are the chances that actually works on the physical robot?
- [44:27]
Matthew Melchano
Yeah, it's interesting. That's actually evolved a lot. So in the early days, we used to think of it as the main purpose of simulation was to show you if it didn't work in simulation, it wasn't going to work on the robot, so you shouldn't even run it. Right. And simulation was good for that purpose, which is a really useful purpose. Right. To save you the time of running something that doesn't work. However, now we're at the point where for many of our mobility behaviors, we can actually run the behavior initially in simulation, and then the robot can just run that behavior and it will, like, work out great. Right. And that's especially true on a lot of our. Like on our E Atlas project. Right. That team has gotten tremendously good sim to real, as they call it in the industry, results around mobility. The area of manipulation, which has sort of a different set of things happening in it, is, I think, still an area that sim to real can be challenging. And so, you know, I think that continued work is happening there for us and I'm sure for others as well, because there's just lots of things that, you know, can be difficult to model. And, you know, a simulator is only going to tell you what you've chosen to model in that simulator. Right. It doesn't. It doesn't Magically disclose new physics to you. Typically, typically you have to be the person who's like, well, I understand how to model this contact interaction or model, you know, this particular, like, object, you know, that can be kind of its own thing as well. So. But great results so far in mobility.
- [45:45]
Sean Falconer
Is all that simulation software is that, that's all built and owned in house.
- [45:50]
Matthew Melchano
We use third party packages, so we use Gazebo and Majoco both in house. And, you know, we have had our own custom simulators in the past and probably will again in the future. And so, you know, it's, it's, we go and look for whatever tool we can find that will move us forward. Right. It's not a key product for us. Right. You know, it's a product enabler. And so we're happy to use whatever capabilities will move our products forward and, you know, be as efficient as we can around it.
- [46:15]
Sean Falconer
You know, earlier on you, you mentioned how like the holy grail is kind of this, you know, being able to build like a general purpose robot. Like, how far away do you think we are from that and what is sort of the main barrier to being able to achieve it?
- [46:28]
Matthew Melchano
You know, humans can do a whole lot of things. And so I think we're actually pretty close to the first humanoid robots being useful in real environments. Right. And I think Boston Dynamics has a tremendous background to go and achieve that goal.
- [46:42]
Sean Falconer
Right.
- [46:42]
Matthew Melchano
You know, we've got tons of practical robotics experience with our spot robot. And so I think that we're very close to essentially demonstrating humanoid robots doing a really useful job for people in some environment. I think that I'm hoping in our lifetime, in my lifetime, I'll see, you know, robots moving among us as sort of like collaborators in the real world, broadly. You know, one of the greatest things about my time at this company has been when I started to see spot robots, robots out in the real world just by themselves, you know, someone else operating it, it's not the company running it, it's someone using it for a purpose or like demoing it somewhere. And that's just been like such a great feeling in robotics that, you know, I can't wait to essentially see that same feeling happen with humanoids.
- [47:23]
Sean Falconer
You've been there 21 years, I believe you said, like, I guess, like, what's kept you engaged and excited through, you know, over two decades of work on, you know, for the same company.
- [47:33]
Matthew Melchano
Yeah, well, I mean, robotics is super interesting, so that's one way to think about it. There's just so much to learn and so much to do. And the field has changed a ton in that time. You know, what we've done as a company has changed a ton in that time. The earliest days were, like, late nights. You know, I would be soldering wires or, like, disassembling things. I'm a software engineer, so that's not my natural habitat, you know, just trying to, like, make these crazy machines work, right? And now, you know, we're shipping products and essentially making, like, robots do things that no one could have imagined. You know, people thought it was crazy to go a legged robot when we built Big Dog. And now, you know, we're building robots that do backflips and, like, work in all these industries, which is a tremendous feeling. And, you know, there's just. In the process, there's just been so many things to, like, learn about. Software, in my mind is like an applied science, right? It's a thing where, like, you could spend a lot of time learning about software itself, but some of the richest stuff you can do with software is go learn about someone else's problems and come away with, like, you know, this sort of knowledge and expertise about a problem that's, like, parallel to your own, but with the precision that you learn by being a software engineer. So, you know, you know how to talk very precisely about problems and cases and all that kind of thing. And, you know, it's just, like, so rewarding to go and, like, learn that in a growing and interesting field, right? And I think that's something that software engineers like, just. That's one of the great things about the job, right, is you get to, like, learn all these applied areas and you're like, yeah, I write software, but I'm secretly an expert in, you know, like, motor control dynamics, right? Or I'm an expert in camera processing. And it's like this total separate area that you get to a bunch of physics for and all that. And robotics is just full of that because there's so many different angles. Also, you know, it's just been. I've been very lucky to join Boston Dynamics when I did, and, you know, it's. We've just been doing really interesting work, and it's just really rewarding to see the impact it has on, like, kids. You know, they have our robots in museums and we bring kids in to see the thing. My brother's children love it. You know, it's just like. Like, you're like, wow, I like, I'm making the world a better place by doing this.
- [49:31]
Sean Falconer
So, yeah, there's definitely. And I talked a little bit about this at the beginning, how I think a lot of people go into engineering. Their sort of first gateway to that is something where robotics, there's a certain fascination because it's. It's like an actual, like, thing that you can touch in physical, which is so different than, you know, maybe a lot of things that we deal with in sort of digital software, which you build an appreciation over time. But as a small child, maybe it's like there's less appreciation there for, I don't know, your business software than something that's, like, a physical robot that's, like, walking around. Awesome. Matt. Well, as we start to wrap up, is there anything else you'd like to share?
- [50:04]
Matthew Melchano
I don't think I've got anything else, just that it was a great opportunity to come and talk, and I hope the topic is interesting to your listeners. And thanks so much for taking the time to have me on the show.
- [50:14]
Sean Falconer
Thanks for coming on. I feel like we kind of just scratched the surface here. We could probably spend another episode or two discussing various topics, so hopefully you'll come back down in the future.
- [50:24]
Matthew Melchano
Great. I'd love to.
- [50:25]
Sean Falconer
All right, thanks. Cheers.