Podcast Summary: Boston Dynamics with Matt Malchano Software Engineering Daily – Episode Released on November 5, 2024

Introduction

In this episode of Software Engineering Daily, host Sean Falconer engages in an in-depth conversation with Matthew Melchano, Vice President of Software at Boston Dynamics. With over two decades of experience in robotics, Matt shares his journey, insights into robotic software development, and the evolution of Boston Dynamics' product lines. The discussion delves into the unique challenges of building software for physical robots, the iterative development process, and the future of general-purpose robotics.

Early Fascination with Robotics

Matt Malchano's passion for robotics began in his early childhood. Reflecting on his journey, Matt states:

“When I was young, actually, like essentially like in elementary school, I just became very excited about robots. [...] I decided when I was going into my master's that I wanted to shift gears and learn about an applied area, right? Like learn how to apply software to robotics.”
(01:33)

His academic path in computer science seamlessly transitioned into robotics, leading him to Boston Dynamics during the pivotal "Big Dog" era, fulfilling his childhood dream.

Software Development in Robotics vs. General Software

Sean Falconer probes into the distinctions between developing software for robots and traditional digital software. Matt elucidates:

“One of the things that separates, I think robotics from just general software is it's like you run on a computer that might not be there some of the time. [...] you'll discover like you'll get a chance to run your algorithm and then you'll go, oh, whoops, the robot's gone for the next day.”
(02:36)

He highlights the inherent unpredictability and non-determinism in robotics, contrasting it with the repeatable nature of general software applications. This unpredictability necessitates efficient software execution to maximize learning from each experiment.

Testing and Resource Management

Discussing the challenges of testing with limited physical robots, Matt explains:

“In the early days of that project, we only had a few robots that we were able to use to build the product robot out of. [...] Is the software bug buggy or is it the hardware that's broken?”
(04:49)

The scarcity of hardware resources in research phases demands meticulous management to distinguish between software and hardware issues, thereby optimizing the testing process.

Purpose Behind Advanced Robotic Behaviors

Falconer raises a common question about the necessity of dynamic behaviors in robots. Matt responds:

“Robots that walk just have a lot of advantages as a product over robots that don't walk. [...] the holy grail of robotics is general purpose robotics.”
(05:58)

He emphasizes the practical benefits of legged locomotion, such as navigating complex environments and manipulating objects, which position Boston Dynamics' robots as versatile tools across various industries.

Challenges in Designing Natural-Like Locomotion

When asked about the difficulties in mimicking natural walking, Matt shares:

“We feel really good about the performance of the legged mobility that our platforms have right now. [...] we don't think of mobility, like legged mobility as that hard.”
(08:28)

While Boston Dynamics has achieved significant milestones in legged mobility, ongoing enhancements like reinforcement learning enable robots to adapt to challenging terrains, such as slippery floors, further refining their movement capabilities.

Software Reusability and Transferability

Sean inquires about the transferability of software skills across different robotic platforms. Matt explains:

“Robotics is a pyramid, right? [...] And so the more you get that pyramid correct, the better off you are the next time you go and build the next behavior.”
(11:54)

Building foundational software components allows for the efficient development of complex behaviors, enabling engineers to leverage existing frameworks when crafting new functionalities.

Reinforcement Learning and Behavioral Control

The discussion shifts to the role of reinforcement learning (RL) in robotics. Matt states:

“Reinforcement learning, which is the thing that you're describing there, [...] we do a mix.”
(13:43)

Boston Dynamics employs a hybrid approach, integrating classical controls with RL to develop robust and adaptable behaviors. While RL offers significant advantages, it remains challenging to fully productize due to the necessity of reliability and consistency in diverse operational scenarios.

Boston Dynamics' Product Lines

Matt outlines Boston Dynamics' three main product lines:

1. SPOT:
   A versatile quadruped robot designed for industrial asset inspection, equipped with various sensors and payloads for tasks like facility monitoring and patrolling.
   “It's primarily designed to do industrial asset inspection.”
   (17:36)

2. Stretch:
   Tailored for warehouse operations, Stretch features a robotic arm with a vacuum gripper for tasks such as unloading trucks and handling inventory.
   “Stretch is planned to be added to Orbit also contains its own tablet controller.”
   (17:36)

3. Electric Atlas:
   The latest addition, Electric Atlas, targets real-world applications and collaborations, notably with Hyundai, focusing on enhancing mobility and functionality for diverse use cases.
   “It's intended for real world applications and we're planning to be testing it along with Hyundai.”
   (17:36)

User Interaction and API Integration

Falconer asks about user interactions with robots like Spot. Matt details:

“Our primary mechanism is a product called Orbit, which we also sell. [...] we're focused on finding ways to deliver robots out to not just people who are robot experts, but to people who can just use them to make their jobs easier.”
(19:28)

Orbit provides a web-based interface for fleet management, scheduling missions, and monitoring robot activities. Additionally, robots come with tablet controllers for manual operation and comprehensive APIs that allow users to program custom behaviors, akin to advanced robotics platforms like Lego Mindstorms.

Development Process and Iterative Improvement

Exploring the development lifecycle, Matt highlights Boston Dynamics' iterative "build it, break it, fix it" philosophy:

“Often it begins with the previous robots. We're hugely iterative. [...] Everything is just being incremental”
(22:35)

Starting with existing hardware and software frameworks enables rapid prototyping and continuous refinement. The transformation of Spot from a lab prototype to a commercial product involved securing software systems, enhancing user interfaces, and ensuring reliability for end-users.

In-House Hardware Development and Reliability

Discussing hardware responsibilities, Matt notes:

“A lot of our hardware is built in house, so certainly a lot of our prototype hardware gets built here. [...] constantly are trying to understand, like, you know, what, like, why is the hardware acting the way it does.”
(26:34)

Boston Dynamics maintains significant in-house hardware capabilities, allowing for comprehensive maintenance and swift resolution of hardware issues. This control over hardware fosters a cohesive integration with software, ensuring higher reliability and facilitating the "break it, fix it" cycle.

Safety Controls in Robotics

Safety is paramount in robot deployment. Matt explains:

“We have an entire safety department inside the org right now. [...] we're starting to look at things from functional safety up to AI capabilities that'll make sure that the robots are able to like work with human beings.”
(32:37)

Boston Dynamics implements rigorous hazard and risk analyses to define operational domains, incorporating both mechanical safeguards and usage protocols. Examples include speed limitations and confined operational areas to prevent unintended interactions and ensure human safety.

AI Integration and Performance Optimization

Balancing real-time responsiveness with computational demands, Matt shares:

“The easy answer is lots of optimization, right? [...] It's really a systems problem.”
(34:39)

Robotic systems employ multi-tiered decision-making processes, segregating high-frequency control loops from slower, more complex AI-driven tasks. Continuous code optimization and distributed processing enhance performance, ensuring timely and efficient robot responses.

Sensor Integration in Boston Dynamics Robots

Matt outlines the diverse sensor suites utilized:

“There are various kinds of sensing on the robot. [...] gives some sense for the kinds of sensing the systems have.”
(36:19)

Robots are equipped with proprioceptive sensors for joint and force measurements, inertial measurement units (IMUs) for body orientation, and diverse perception sensors like stereo cameras and lidars. Additionally, modular payloads allow customization for specific applications, enhancing environmental awareness and operational capabilities.

Programming Languages and Software Stack

Exploring the software ecosystem, Matt states:

“We're hugely iterative. [...] a lot of languages.”
(37:55)

Boston Dynamics employs a polyglot programming approach, utilizing languages such as C for low-level controls, Python for scripting and analysis, TypeScript and JavaScript for web interfaces, and emerging languages like Go and Rust for specialized tasks. This diverse stack supports the integration of various functionalities across different robotic platforms.

Software Engineering Department Structure

Matt provides an overview of the software team:

“Boston Dynamics is around 250 software engineers, [...] all of those things are part of the group of people who are building software at Boston Dynamics.”
(39:07)

The engineering department is segmented into product-focused teams for each robot line and a central team dedicated to engineering enablement and R&D. The team composition spans robotics specialists, customer-focused engineers, web developers, firmware experts, and safety software developers, fostering a multidisciplinary environment.

Open Source Technologies and Frameworks

Highlighting the use of external resources, Matt notes:

“We use a lot of open source software. [...] we tend to be shy about committing to frameworks.”
(40:20)

Boston Dynamics leverages numerous open-source packages such as OpenCV, the Point Cloud Library (PCL), Eigen for linear algebra, and Google’s Protocol Buffers and gRPC for communication. While they utilize these tools extensively, the company remains cautious about over-reliance on specific frameworks like ROS to maintain flexibility and control over their software roadmap.

Backwards Compatibility and Remote Updates

Addressing software maintenance, Matt explains:

“We design all of our robots to accept remote updates. [...] everything from tip of master.”
(42:46)

Robots receive remote updates to ensure consistency and security, maintaining compatibility across software versions and hardware generations. Boston Dynamics employs robust policies for API versioning and deprecation to minimize compatibility issues, streamlining the software maintenance process.

Simulation vs. Real-World Testing

Discussing the role of simulation, Matt states:

“That's actually evolved a lot. [...] can run that behavior and it will, like, work out great.”
(44:27)

Initially viewed as a fail-fast mechanism, simulation has advanced to the point where mobility behaviors developed in simulation reliably translate to physical robots. While sim-to-real transfer for mobility has matured, manipulation tasks still present challenges, requiring ongoing refinement to bridge the simulation-reality gap.

Path to General-Purpose Robotics

When exploring the future of robotics, Matt envisions:

“You know, humans can do a whole lot of things. [...] someone using it for a purpose or like demoing it somewhere.”
(46:28)

While specialized robots excel in predefined tasks, Boston Dynamics aims to bridge the gap towards general-purpose robots capable of performing diverse functions in dynamic environments. Matt anticipates the integration of humanoid robots into everyday settings, collaborating seamlessly with humans and enhancing various aspects of daily life.

Personal Motivation and Longevity at Boston Dynamics

Reflecting on his long tenure, Matt shares:

“Robotics is super interesting, [...] it’s just an amazing feeling to be like, wow, you know, I wrote that on some field test out in the Mojave desert.”
(47:33, 42:25)

Matt attributes his sustained enthusiasm to the ever-evolving challenges and learning opportunities in robotics. The ability to witness decades-old software still in use today underscores the enduring impact of his work, fueling his passion for advancing robotic technologies.

Conclusion

Matt Malchano's comprehensive insights highlight the intricate interplay between software and hardware in robotics, the strategic development of Boston Dynamics' product lines, and the company's commitment to innovation and reliability. As Boston Dynamics continues to push the boundaries of what robots can achieve, the conversation underscores the transformative potential of robotics in industrial applications and beyond.

For more information about Sean Falconer's work, check the show notes. Stay tuned for future episodes where Matt Malchano may return to delve deeper into the fascinating world of robotics.