← Software Engineering Daily
Software Engineering Daily
The Hardware Bottleneck AI Can’t Fix
00:50:31
Software engineering has developed powerful tools for observability, data management, and continuous testing, but hardware engineering has largely not kept pace. The feedback loops, tooling, and infrastructure that software engineers take for granted s...
Loading summary
Transcript62 lines
- [00:00]
Narrator
Software engineering has developed powerful tools for observability, data management, and continuous testing, but hardware engineering has largely not kept pace. The feedback loops, tooling and infrastructure that software engineers take for granted simply do not exist in most hardware programs. Nominal is a data platform built to help hardware organizations move at the same speed as software teams. It manages the hardware data supply chain end to end, from ingesting high frequency sensor data off physical assets to an enabling real time control room monitoring, post test analysis and simulation correlation. Jason Hawk is the co founder and CEO of Nominal, and he has a background spanning distributed data systems at Palantir and cloud infrastructure at Vercel. In this episode, Jason joins Kevin Ball to discuss why hardware engineering has lagged so far behind software in tooling and observability. The unique data challenges of working with high frequency time series sensor data, how Nominal handles both real time control room workflows and post test analysis why AI agents are transforming software development but have not yet made the same leap in hardware and what it would take to close that gap Kevin Ball, or K. Ball, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow K. Ball on Twitter or LinkedIn or visit his website K. Ball LLC.
- [01:49]
Kevin Ball
Jason, welcome to the show.
- [01:51]
Jason Hawk
Thanks for having me Kevin.
- [01:53]
Kevin Ball
Yeah, I'm excited. It's going to be fun to talk with you because the subject area of hardware and hardware testing is like a slant for me. It's like real close to what I do, but it's not what I do. So I'm excited to dig in. But let's start with you. Can you give me a little bit of your background and like how you got into what you're doing today with Nominal?
- [02:11]
Jason Hawk
Yeah, I'll give you kind of the longer story, which is that I started my life as someone who was more of a mathematician than an engineer. I think software was a way for me to like enter engineering and I got like I fell in love with distributed data systems. I built a lot of software at Palantir early in my career, also fell in love with working for like really technical users, people who had expertise but not necessarily software and data systems. And Nominal for me is like the culmination of some things that I'm really passionate about. So it's growing software teams and products from scratch, but also with like big ambitions. Meeting those really technical users. In our case it's like hardware engineers, people who have obsessed over electrical systems, aerospace systems and then like frankly just solving like really hard but fun like software problems. So full stack up and down, like how do you store petabytes of high frequency sensor data all the way up to like the minutiae of the interactions that people expect out of our front end products? Because you have to consider those when you're thinking about how the data is stored and moved around. So I've been doing that over the course of my career. I love tooling, I love infrastructure. I've been lucky to be part of some like hypergrowth organizations. I'd spent some time at Vercel working on their cloud infrastructure. And how do you make very low latency web experiences for people across the globe. That team was very fun. It was four people when I joined and we grew to 40 in under nine months. So in many ways not my first rodeo, but gotten to do some different things over time.
- [03:31]
Kevin Ball
Awesome. So let's maybe start with a quick overview of what nominal does for folks and then we can dive into the technical guts of it.
- [03:39]
Jason Hawk
So we try to help hardware organizations move at the speed of software and we think that the right way to do that is to build a connected suite of software products specifically focusing on managing data. And we talk about the hardware data supply chain. So there's a physical sensor somewhere in the world and fast moving modern hardware programs often need to get that data in front of many, many different eyeballs. And they're working with really high frequency multimodal data. So you can imagine a million data points per second are being produced on something like a test aircraft and you want to correlate that with video or you might have some sensors or other data sources that are on the ground while the airplane is in the air. And getting that all to work well together is like a software problem. It's a human and a process problem. And as software engineers we've built like amazing tools for ourselves. We're very used to like an elastic cloud where you can run like INF infinite, cicd. And the observability that we are used to is just not there in the hardware world. Some companies like SpaceX and Tesla and then Anduril, one of our customers have done really good stuff here. But there's this wave of smaller companies that are trying to be really innovative in hardware and move fast and they really need help from software engineers and people who have spent a lot of time there.
- [04:52]
Kevin Ball
Most of the folks listening to this we are software engineers and we know maybe what this looks like in the software industry, but have no idea what the state of the art is or what the norm is in, in hardware engineering. So when you say like helping hardware move at software speeds, like, what does the traditional hardware speed look like and what are the pieces that go into that?
- [05:12]
Jason Hawk
Yeah. So when my co founder Bryce and I were getting to know each other and he was teaching me about like what he had done, NASA mission operations. So like we have sent a probe, we've spent over a billion dollars of taxpayer money to Jupiter. Like, what is it like to deploy software procedures to that probe and more importantly, like, test them ahead of time to make sure that you don't like, crash or destroy the mission. And my mind was like, immediately, like, I started geeking out because I realized that when I think of cicd, I think of these. Like, again, like you're doing a unit test, maybe a more complex integration or end to end test would take minutes. But when you're talking about like something like a satellite, an airplane or a spacecraft, you might spend a million dollars on a test bed, like a physical table where you're like arranging batteries and wires. And you need to pre test before you test on that test bed. Because if your test bed breaks, you might only have one or two of them because they're so expensive and your whole organization grinds to a halt. And so like the heterogeneity, I always have trouble saying that word, but it's like, it's not homogeneous the way you kind of do this. You know, something like a CI test pipeline. And so immediately there's like a lot of need for like, interesting tooling there, but just to kind of like ground the conversation, I always like to talk about test aircraft. They're really easy to talk about. But you know, a lot of our customers, they fly something in the air, it lands. You need to get the hard drive off of the airplane. You need to somehow get that data to an engineer who designed the system if there was an issue and you need to kind of debug it. And you know, some of our customers, that process could take on the order of like, literally like a day or two versus we're used to that feedback cycle being like so much tighter. As software engineers.
- [06:46]
Kevin Ball
Yeah, it's wild to think about, like to get your test data back, you have to literally transfer the hard drive and then to run another test, you're either flying a plane again or you're configuring some sort of hardware system. Okay. So with that as like our grounding statement, right? Hardware has these like physical constraints. What can you actually do with software to make that better?
- [07:11]
Jason Hawk
Yeah. So again, just to bring us back to that example of like the airplane lands and now you're getting the data off of it. A really common stumbling block we had seen is that the schema for the data that you're taking off of the hard drive and now maybe trying to get into some kind of standardized long term format, it can change as the software you're deploying to the aircraft itself changes. And in the field you might have someone who's really tired, they're trying to get home at the end of the day. And they're not a software engineer, they're not even an engineer who designed the physical system. They're an operator or technician. And they might forget to change a config value in their ground control software when they, let's say they downgrade the software because there's an issue now, you need to downgrade the configuration and if you don't, the data gets garbled and all of a sudden like, you know, you're back to this like really hard human process of ever trying to recover it. That's the kind of stuff that we kind of realized was happening at some of our customers. Like one of them said, hey, all of the data ends up in this, like network storage, like, help us dig in there. And we realized that only 10% of it even got to that point because of these kind of like earlier pain points. But like, all of that is something that we think that we can help with. So in that example I just gave, we actually built a better software tool for the test operators. And then they didn't even have to think about this. But one of the side effects was that that config value was getting automatically updated because we were syncing the version that's running on the aircraft with the version of the parser that was running locally. And then instead of them having to upload a file and kind of do this process, like again, that was happening in the background. That's like a more clunky example, I think. Same organization, but now let's say they're installing a Starlink receiver and transceiver on the aircraft. Like all of a sudden they're going from a. The data is getting ripped off of a hard drive at the end of the flight to it is Streaming. Welcome to 2026. But that can be a really hard architectural change to navigate if your software resources are focused on the guidance and Navigation and control of an aircraft, not the data processing. But we help tons of customers through that transition. Our architecture is designed to handle it. And it's a fun and interesting software problem to solve, but that's the kind of thing that we can work with them on.
- [09:20]
Kevin Ball
Yeah, it's fascinating because you're almost having to be like vertical integration across all these different levels of like from the. Just making it easy to collect the data at all. Right. The automated configuration change. Suddenly like the data is actually getting through without a person doing that to like, hey, you can actually stream this data. Like, let's help you handle that. Completely change the way you think about data. So I guess the question would be then, what are the problems that you end up having to. Like you said, we're going from essentially batch hard drive data to streaming data. You have to support that whole gamut. What does that end up looking like in your architecture? Like, what did you have to do to do that? And what does the end user perceive as they make that transition?
- [10:06]
Jason Hawk
So I would say that as with any startup we've had like over time, had to pick and choose which things we handle really well and which things we kind of do a passable job at so that we earn the right to do like a better job down the line. So our original streaming architecture, like I'm going to be honest, was like pretty weak. We kind of built a proof of concept that we could bring to our customers and open their eyes to like, hey, you are thinking of post test data analysis in a certain way. But we actually think it could be this new world where you're kind of doing the analysis on the fly, some of it's automatic while the aircraft is in the air. And then one of our customers said, we actually want to have nominal shown in our control room. And it's going to be like a human is looking at a dashboard and there's an aircraft and there's a person in that aircraft and we want to know very, very quickly if a safety check is failing. And we surged on that as an engineering team because we were pretty confident we could get the latency to be within those bounds. And, and I guess it was a happy accident that the more analytical user interface we had built was other users were starting to look at it and seeing it as more of like a monitoring dashboard. And I would say like, now we're in a place where like those two use cases are both supported and like very crucial to our customers. So again, it's like this. Imagine you have a control room like you could think of it like Houston Mission Control, like you're launching something or there's some kind of test and there's a lot of humans looking at a lot of screens. And then there's the very kind of like you're a propulsion engineer who's digging through hundreds or thousands of flights trying to find kind of statistical patterns. But from an architecture standpoint, like, we've had to kind of invest in like, you can think of it as like a hot path and a cold path that need to stay in sync. They can't be telling different stories. But that really, really low latency pipeline and the more cold storage or I need to do OLAP style analysis. We support both of those and or trying to almost abstract away from our customers and our users. The fact that from a software standpoint, those are really very different. We want them to write the logic once and be able to kind of pivot to the different use cases seamlessly.
- [12:06]
Kevin Ball
It's interesting because like in one of those cases you are in a traditional development QA cycle or postmortem cycle where you're like, something went wrong or something happened. I want to look at it differently. And the other one is sounding to me like live observability. Like, let's have health dashboards. Let's have, you know, go no go decisions happening.
- [12:26]
Jason Hawk
Yeah, it's crazy the amount of data that people will cram into one nominal screen. Like, we've had to adjust parts of our user interface because we didn't necessarily predict like, oh, it looks more like an Excel spreadsheet than a web ui. But it's just because these systems are really intense and the data density is like, becomes very valuable. It's kind of a classic lesson in kind of like user interface. But if you have an expert user who's using this thing day in and day out, it's very different than like, you know, I'm designing a web UI that someone might only, you know, there's a funnel that I need to get them through and it's the only time in their life they're going to perform it. And then I think just like to highlight some of the software fun of this is like we deploy off cloud, so like there's customers where they need everything to be running locally. And I can talk about real time. That's like a interesting topic for us to get into later, but without getting into real time. Like for these low latency control room workflows, you might not want to have a network hopped AWS involved, but there's customers that, you know, they're using Starlink, they don't want to necessarily administer their own stack of data software. And so they are using AWS and for us to get data going to AWS and then rendering back into a react application, all with like very high volumes of data, with like tight SLAs around latency, like that's a really hard software problem that we've invested a lot in and we think it makes sense for our market.
- [13:41]
Kevin Ball
So kind of going into that, you know, the volume of data here and thinking about, well, I guess maybe first let's start with like what does the data shape look like? Right, because when we're talking about observability, this isn't stack traces and like post hoc things, this is something else going on. So what is the shape of the data you end up working with?
- [13:58]
Jason Hawk
Mostly, yeah. So if you think of a satellite or an aircraft, especially one that's in test operations, there could easily be 10,000, 20,000 sensors on that asset. And each sensor might be kilohertz, might be 10 kilohertz. Sometimes we see megahertz sensors. So that's thousand, ten thousand or million points per second. And depending on the test that's being performed, you might never want to drop a data point. So the example I always give is like if you're building a rocket engine as a startup, you know, you might spend years and tens of millions of dollars getting to the point of doing your first rocket fire test. And even if that rocket fire only lasts eight seconds or something, but like you never want to throw any of that data, that's your entire company is essentially that data asset that you got from that test. And that is super different from software observability, where you tend to forget everything a week later. You're aggregating things into 10 or 15 second buckets and getting some kind of stats on them. You're thinking about things like P99. I always joke that like as a software engineer I was trained to think of my infrastructure like cattle, not pets. But for our customers everything is a pet. Kind of like they'll have like one or two aircraft that the entire company is oriented around until you earn the right to kind of scale up production. But yeah, so it's tons of time series data. That video feed can be really important and doing frame precise correlation between video and time series is actually like really, really hard. That's actually, I'll be honest, we like underestimated the challenge of that at one point in our journey. And I'M like very grateful to our we have wonderful software engineers at Nominal and you know some of them became experts in like video encoding and like how do you actually build a video stack? Like a lot of open source software is not designed for that framework seeking use case. It's designed for like social media, video playing or something like that. So that's the types of the, that's the shapes of the data we see. And then I'm just trying to think of like I already kind of talked about this like network issue but like you might have radio data, you might have the data that is coming off at the end of the test off of a hard drive, you might have the streaming data. Getting those all to kind of like tell the same story at the end of the day can be a challenge.
- [16:04]
Narrator
You know, Fidelity is a financial services leader, but did you know that inside Fidelity is a community of technologists working together to shape the future of finance and tech. Fidelity is always investing in tomorrow. From emerging tech to cutting edge tools that will transform what comes next. Their technologists are encouraged to keep learning so they can expand their skill sets, explore new ground, and stay ahead of this rapidly evolving industry. And right now, Fidelity is hiring technologists to join their team. Fidelity technologists get the best of both worlds. Startup energy that's grounded in the stability of a financial institution. That means support, resources and amazing benefits. Bring your skills to a culture where you're empowered to dream big and build the tech that drives an organization and makes a real impact on people's lives. Find out more@tech.fidelitycareers.com that's tech.fidelitycareers.com Fidelity is an equal opportunity employer.
- [17:05]
Kevin Ball
TurboPuffer is how companies like Anthropic, Cursor,
- [17:08]
Jason Hawk
Notion, Atlassian, and Ramp ship their most ambitious search features. TurboPuffer is a serverless vector and full text search engine built on object storage. It's up to 95% cheaper than traditional search databases and just as fast. With TurboPuffer, you can index and search 50 million documents at 10 millisecond P90 query latency for less than $100 per month. Head to turbopuffer.com sed to get your first month free.
- [17:35]
Kevin Ball
If you're running Postgres in production, you've probably felt the moment analytical queries start fighting your transactional workload. Most teams end up adding a second database and all the pipeline complexity that comes with it. Tiger Data, creators of TimescaleDB, takes a different approach. We extend Postgres with hybrid row and columnar storage, so one table handles both writes and analytical scans. Native compression cuts storage costs up to 95%. Continuous aggregates keep dashboards live without bash jobs and it scales to petabytes without you re architecting. Companies like Cloudflare, Octave Energy, Schneider, Axpo and Flowco run production workloads on Tiger data today. No stale data, no second system to operate, just postgres managed for you. Ready for the workload you're building toward? Try it free@tigerdata.com I want to dig in a little bit to what are the differences implied by these companies that are working with pets rather than cattle? Right. Like where everything is tied into this one physical thing or two or three physical things that may be subtly different. Right. They may be nominally working towards the same end point, but like each one has substantial changes. So like, are there versioning implications? Like, what does that do to the development process?
- [18:46]
Jason Hawk
Yeah. One of the interesting stories I heard recently is this one company, when they detect a problem in a component, they want to trace it back to which 3D printer printed that component. And they literally named the 3D printers. And it's because the other ones printed by that printer might have had issues around that time. And so it's this really interesting cataloging problem. And at nominal, we talk a lot about the hierarchy of assets, but you might have two aircraft, and each one might have two engines, and at some point you might actually like swap out one engine from one aircraft to another. And I'm used to thinking about kubernetes and your nodes and your pods and things kind of get shuffled around all the time and. But like understanding which engine, was it engine two or engine three that was on the left side of aircraft 100A last week, like that kind of stuff is make or break. And it can be really, really hard to manage. And I think some organizations, you know, they do have to figure out how to treat some things more like cattle, like pets, for their own sanity. But the dollar numbers are just like so different. Right. If you launch a satellite and it fails, like, that's months of timeline and tens of millions of dollars. And so you're, you're willing to double check things a lot more than I think people in the software engineering world are used to.
- [20:01]
Kevin Ball
Cataloging is kind of an interesting domain. So is that something that you do as a part of your software as well, or you provide them tools to integrate their catalog with the test results? Or like, how does that end up working?
- [20:11]
Jason Hawk
Yeah, one of the things that's really crucial to our data model is this concept of events. And so we are basically, through our user interface, encouraging people, if they identify a region of time where something has happened, to tag that as an event. And then it's available to anyone else who's looking at that. They could be looking at a different subsystem, different set of sensors, but still be aware that something was going on in the overall system at that point in time. And then a lot of these organizations, you know, you as the responsible engineer who's designed a system, a test happens. You want to be like yourself, eyeballs looking at the data feeds to kind of understand did things perform as expected or is there something anomalous happening? And then over time, you want to kind of like encode those into automated checks. So you're kind of like writing code or lightweight logic. By the way, like, a story I always tell is that, you know, from the earliest days of our product, you could take these raw sensor feeds. Like I was saying, kilohertz sensors, you have 10,000 of them and you're doing math on them. Like, these people, you know, they do physics every day. I haven't done physics since I was in undergrad and makes me sad. I miss it. But I asked them, this is a satellite customer that we were working with early on. I asked them, like, you know, what. What type of mathematics do you need to do in our platform? Like, we support, obviously, basic arithmetic, but we're trying to understand how wide our library scope needs to be. And they were like, oh, nothing too complicated, just quaternion operations. And they weren't being.
- [21:31]
Kevin Ball
Oh, is that all?
- [21:32]
Jason Hawk
That's all? Yeah, just, you know, and it makes sense if you're doing guidance, navigation and control for a satellite. That's kind of the table stakes. And if anyone has used Matlab, like, you know, that is a useful thing to have in the back of your mind when you're thinking about the capabilities of our platform. And our goal is not to 100% replace MATLAB in an organization. Like, there's always going to be certain types of modeling or calculation or custom code that our users need to write. But, you know, maybe the one person who's writing that Matlab script can package that up and it can be running in, like, orchestrated by our platform. And so the other 200 people who might need to take advantage of that logic, they don't themselves need to open up Matlab or even definitely not need to understand the code. I forgot the original question.
- [22:14]
Kevin Ball
I was just looking for how much of the categorization. So it sounds like from Event standpoint. Right. Like you can correlate events. Right. Which makes sense. Right. If you're looking at time series data and you say, oh, something went wrong in, I don't know, this engine over here, that's probably going to impact a bunch of your other sensor time series. And so you want to be able to see that. But you also mentioned this example of tracing back the hardware to for example, the 3D printer that printed it. It made me think of data lineage problems. Right. It's not a data lineage problem, it's like a component lineage problem. Right. This piece was printed here and assembled here and connected here and like being able to track all that back. So I was curious if that's something that your, your software is supporting or that's a third party thing that they're doing independently or how does that work?
- [22:58]
Jason Hawk
You're right on the money because I, I always talk about data tagging as well. So yes, it's cataloging. It is tagging. It is, it is very similar to data lineage. You might have your left wing and your right wing, or you will have these arrays of sensors that are themselves like modules that you're printing multiple times onto your large system that you're developing. And you want to kind of develop logic for checking each of those subsystems and then you want to repeat that logic, kind of like a software program. But doing that is really hard because you have to have a schema to your sensors that allows it to be possible that schema will change over time. And it's the types of things that again, like people who are building for a specific software system are used to solving those problems. But we are trying to build a generic platform that can work for someone building an aircraft or someone building a nuclear reactor. And so we think about, you know, tagging of time series data in such a manner that you can make those composable, you can have logic, you can compose it, and then you can apply different things. And we think the cataloging is really important and some of the work we do with our customers is helping them, whether they're like a five person startup or a, a 5,000 person organization that's starting a new program. Like how do you be thoughtful about that cataloging from the beginning? Really powerful example. But I think it's really grounding is if I am doing a test and something goes wrong, I want to compare it to a simulation that we ran that's supposed to be corresponding with this test. How do I summon that data and overlay them on top of each other? Really easily. If you've tagged your data well, that's just a couple clicks if you haven't tagged your data at all. And it's just kind of in a big soup. You're kind of writing a lot of custom SQL. And that's what we're trying to get our users away from is the same way that I use datadog and the datadog Agent is cleverly tagging many aspects of my data based on the node and the POD and the other infrastructure things that it can introspect. We would love to get our users to the same place. And so for us that means investing more in the ingestion agent code that's frankly like running closer to the edge or even like on hardware itself to try to make that a zero cost abstraction for our hardware engineering users.
- [25:01]
Kevin Ball
That's super interesting. So two different dimensions. I want to go down on this based on that. So one I'm going to go down a little bit is. So you mentioned using simulation and comparing simulation and hardware. And that's I think a space that I think it was. Last year I was at a panel that was hardware, it was automotive manufacturers and they were talking about this was one of Tesla's big advantages is that they have all of their stuff in simulation, they can do it now the traditional manufacturers are trying to get more and more of their things in simulation environments so they can iterate and improve faster there. Is that a use case that you are directly supporting? So you're doing your test infrastructure for here, like get the outputs of the simulations and then do it with your hardware, stack the live version later. Or is that once again like an integration point that you're integrating out to something separate?
- [25:47]
Jason Hawk
I think for us right now it's an integration point. So we don't claim to be experts in simulation, but we really do want to make simulation much more leveraged within our customers. If they are running simulations, so many of them run into this kind of chair swivel problem of I need to have three tools up to be able to make the correlations that I want to or worse, I need to be reaching out to another team. I think the world is changing and 25 years ago true hardware innovation was pretty localized to large conglomerates, traditional primes, companies like General Electric that had massive, massive resources. And it made sense for those companies to develop over the course of 30 or 40 years like proprietary best in the world simulation software. But as more of this innovation is moving into smaller, more insurgent companies, like they need to grab some stuff off the shelf and try to piece it together. I think that is itself really frustrating. So where we're trying to, we're trying to meet people as close to where they're at as we can. Hey, if you're using the simulation tool, let's figure out how to get that data in to nominal. Like I was saying, tag it and catalog in such a manner. To us it's just data. Like it could have come from a simulation, it could have come from a hardware test. But we'll oftentimes talk to leaders of test organizations that have this dream of like how we're always simulating and if something goes wrong, we're going back and like learning about how to improve the simulation. But their organizations can really easily not meet those aspirations because of tooling fatigue. So that's what we're trying to solve.
- [27:13]
Kevin Ball
That makes a lot of sense. The other direction I wanted to go down is you talked about having agents running kind of at the edge at the sensor. And I know in sort of software cloud observability, oftentimes a lot of what those agents are doing is filtering out data that is not relevant to send upstream, so they reduce the quantity. But to your earlier point, right, like the amount of data that you want to filter in a hardware context may be much smaller, maybe very different. Particularly, you know, I love your rocket ship example, right? You have 8 seconds of data and that is the value of your company right there. That's everything you've invested to get. So you don't want to throw that away. So what exactly are those edge agents doing? Are they still doing some amount of filtering or like what happens in that case?
- [27:54]
Jason Hawk
Yeah, they are still filtering because of like network limitations and the fact that like Software is not 100% reliable, but they are then having to save the data locally and then figure out a way to kind of like backflip fill later. So that's something, you know, we have a customer right now, they're doing like really intensive testing in a network constrained environment. And it used to be that there was massive latency between HQ headquarters reviewing the data and what was going on at the test site because of that network limitation. And we have now made it so that like I said, the logic can be composed in one place and it can be pushed to the network constrained location. So some of the checks can be happening automatically in a very low latency manner and then the data is getting uploaded at the rate it can. But there's this mesh concept of you might have your HQ and your test site Each have a database and they eventually get in sync with each other again to use the rocket fire example, 8 seconds, tons and tons of data. But they're not doing that constantly. They might only be doing a few times a day. So if you amortize kind of data upload over long enough time horizons you can catch up. But to your question, it's like people who have thought about things like back pressure and distributed systems super, super relevant for our use cases, you oftentimes are having to prioritize between safety critical information, high priority observability and then like kind of everything else. And so there's like a hard drive that you're, you're having to store to and then catch up later. It's not dropping the data, it's just figuring out how to handle it given the constraints.
- [29:23]
Kevin Ball
Nice. I'd love to dig in a little bit to your topic of kind of the way that hardware innovation is shifting. First off, just like open question you said, you know, 30 years ago is all these big companies now it's much more insurgent. What are you seeing in terms of hardware innovation and the ecosystem? How is it changing what's happening there?
- [29:41]
Jason Hawk
Well, it's really interesting because I want to give some credit to some of the larger incumbent players, but like the Air Force being one of them. We worked with the Air force, they employed 20,000 test engineers. Speaking of pets versus cattle, like as taxpayers we are like all shared owners in arc jets and wind tunnels. And these like, they're literally, they're called the national labs. They're these assets that are crucial for testing and, and whether those can operate well or not kind of makes the difference between the US falling behind or staying ahead of certain international competitors. So that is still a thing. And us investing and partnering with those organizations as the software and tech industry I think is really important. But then we have also tons of venture capital, dollar and tons of brain power trying to do things in new ways. And frankly SpaceX has kind of trailblazed here and shown that it's possible. And what's both amazing and terrifying is that these startups are trying to go even faster than SpaceX. Like they're trying to do more with less than ever before. So it's aggressive timelines, it's lean teams. And I think that's part of why we started Nominal is like if every single one of these companies is going to be finding themselves having to set up AWS infrastructure and then build some custom tooling on top of it, that's just not what makes Their beer taste better. Like, we should be partnering with them and helping them do that. And we're seeing that happen. So we have startups where this is not my prediction when I started Nominal, but they want to use Nominal and they reach out to us and we get them spun up and we send them some documentation and they're ingesting the data and running without us even having to go on site, which is amazing. And these are seven or eight person organizations. We have these metrics where we keep track of what percent of the engineers at the company are logging to Nominal on a given week. And for some of the startups, it's 100%. The CEO is logging in because he or she wants to understand how their system is performing week over week. And then I think it's incredible. Like, I don't know if when I was growing up I would have thought of a nuclear reactor innovation being able to be carried out by essentially a startup, but we're seeing that more and more. It's awesome.
- [31:45]
Kevin Ball
No, it's fascinating because I feel like in my career, roughly spanning since early 2000s until now, we went through a period where it felt like all of the fast innovation was happening in software. Software was changing quickly. There was software, software, software. Hardware was slow moving slowly. Not that much change. There was change, but it was like, oh, my car got better miles per gallon now, or things like that. It's not super visible. And then it definitely feels like in the last few years, suddenly we have got private space companies and we have all this drone changes and all this other stuff going on in hardware. That is. It's kind of wild.
- [32:22]
Jason Hawk
It really is. And we have to ground that in. There's still different constraints building and innovating in hardware than there is in software. And that's one of the things that I get nervous about sometimes, especially in the last couple years, where software engineers are having agents write code and the agents can run these evaluation loops and there's this feedback loop that again, it's just like so fast and so intense. And a lot of times the supply chain is the supply chain in the world of hardware. And you can't shave off these kind of months other than by kind of designing things well from the beginning and having an organization that is geared towards iterative development and not kind of like waterfall style development. So if you look at a program like Starlink, they have multiple generations of Starlink design orbiting the planet simultaneously. They're constantly getting data off of those satellites and feeding them into their manufacturing improvements. And that's the way kind of like every mature hardware program like needs to shift to. You reference Tesla and their simulation. Like the amount of effort that Tesla has put into their internal data cataloging, I think would like blow other people's minds. You know, Apple, not an organization that's super public about the way they handle this kind of stuff, but absolutely massive investments in kind of test infrastructure and test data management. But like, the truth is that I don't know if it's like because of 3D printing or just the fact that like, so many people have like grown up inside SpaceX and now they can go off and build their own companies and there's this kind of like culture of confidence and doing things more quickly. But yeah, we're kind of in like a golden era of that innovation. And as a software engineer, I'm a little bit. I get FOMO sometimes. I still love building software. And so I, I joke that Nominal is the closest I can get to contributing to all of this while still myself waking up and thinking about code every day.
- [34:00]
Kevin Ball
So let's kind of go down that. Because you mentioned the coding agents and the coding agent Loop, and that's totally transforming software development at the moment. Right. People are really trying to figure out what it looks like to be a software engineer. Looking out a few weeks or months or I think years is too hard at this point because stuff is changing so rapidly there. You had a public LinkedIn post about it's not yet doing that in hardware. AI is not changing hardware engineering to anywhere near the same degree. Can you expand on that? And maybe we can dig into some of the whys behind that and what it would take to actually accelerate or expand it in that way.
- [34:36]
Jason Hawk
So one of our customers that I can't give specific details about, but they're trying to build a machine that's like the size of a building and it's taking. They're going as fast as possible. And it's frankly the fastest I've seen any organization go. And it's still taking them months. But because of Nominal like this, what their CEO had said is we are looking at data and we are noticing problems and we're catching them earlier. And so it's taking us 30 minutes to fix them instead of something catastrophic happening and us losing two days. So that is like the tightness of loop that it's still like, that's an order of magnitude. Right, that we're providing. But it is very different than like, I write some code and then I run a unit test and all of a Sudden, no, like now the code is writing itself and running its own unit test, which is an order of magnitude, but like a microcosmic scale. I really want, I think as a, as humanity we should have the ambition of getting to that in the hardware world. But that requires this concept of like well, the hardware can be tested in a unit testable manner and that feedback loop can then introduce the opportunity for something like a design agent to understand what's working and not working in a hardware design. And there's aspects of this, right? Like I think there's amazing companies that are, you know, you can kind of give it certain parameters and it will like spit out like a proposed design and then there are simulations that you can run on it. But at the end of the day you still have to like physically build something and wire it up. Maybe you're driving it out to a desert and flying it and like those things just, just take a lot of time and human coordination. There's safety risks involved. And I just think we're like willing to be pretty YOLO with openclaw, but like would not want to with a rocket fire or especially like any kind of flight that has a person inside the vehicle. And so to get there we need to have some place for like the data to kind of be in, be aggregated and for people to kind of give thumbs ups and thumbs down on that data. Talking about the event catalog again. But if you have a sensor that's at a rocket test facility, and again that rocket is only being fired for a few seconds at a time, a few times a week, the rest of the time it's sitting dormant. You don't want to drop any data points. But the eight seconds of tens of millions of dollars of investment, like that golden data asset, you have a lot of dead time in there that you can delete and filter out. Understanding where those regions of interest and disinterest are having the people who got the degrees in physics and mechanical engineering and are like working on top of a platform like nominal and telling us where the interesting data is, like what's anomalous, what's expected, what's anomalous but fine versus anomalous but dangerous. These are the types of data assets that we need to be accumulating if we ever hope to train agents on the world of physical engineering.
- [37:08]
Kevin Ball
Yeah, I think what's interesting, one of the reasons that software is so amenable to agents is because you have that very tight feedback loop and you can give the agent the ability to just try shit and Then see what happens and kind of iterate in feedback because they're probabilistic, they're not going to get it right every time. But if you can do that. So looking at the hardware world, some of what I'm hearing is that feedback loop is harder because there's a human. You got to print a thing and do a thing. Does simulation solve this? Can we get or is simulation not good enough? How do we get to that level of an agent can have a hypothesis, try a thing, get a feedback loop, and then iterate on that without a human having to be in that loop.
- [37:47]
Jason Hawk
This is where we're getting above my pay grade. But it's like the simulation is not good enough. And I almost go back to being a curious college student and thinking about these things. But I start to wonder, is it actually better to send the agents thinking at the level of physics and simulation and building up from there versus trying to give them simulation tools and have them jump in at the hardware design layer? The quip I always share is that if you ask an LLM about the way a jet will perform, it's not thinking in terms of physics. It's kind of thinking in terms of humans who have translated physics into English, and that it is itself composing the concepts of English summaries and kind of synthesizing some results to you. And then you have to go test if the physics end up matching that or not. And that is not good enough to actually make progress. Very different than software where you're working in code, which they natively understand.
- [38:40]
Kevin Ball
So a thing that I have observed is LLMs are systematically bad at a number of things, but one of those things is time. And you talk about you're doing a lot of stuff with time and time series data. Is there an LLM component to nominal software, or do you cut that out and not deal with that at all right now?
- [38:56]
Jason Hawk
Oh, yeah. Thank you for asking that. Because I should be really clear. Like, we released just a few weeks ago, like, all of our customers have access to it now. But you can interact with a workbook agent, you can have a chat interface. And the reason we added that is because it actually reduces a lot of friction for our users because we have to support so many complex calculations and visualizations inside our platform that, I don't know, it can take 100 clicks to build up a visualization of what happened during a flight, doing a flight test analysis. And if you have done this before inside our platform, there are certain equations that we can communicate about in English, and there's enough literature on the Internet about what that means that the agent can get kind of right understanding. Okay, well, speed is the square root of the sum of the squares of the component speeds. You can ask questions like this, and it can kind of perform a lot of clicks for you. So that's something that we've built in right away and we think is really powerful. And then we honestly have a ton of stuff in our roadmap where we think there's more ways to reduce friction in our platform, leveraging these technologies. But then also probably some exciting things we can do in terms of training on time series and using these architectures to help our users identify things that are interesting that even a large organization of 100 smart engineers might miss. Lots of kind of, like, research that we're doing there and excited to see where it goes.
- [40:12]
Kevin Ball
Yeah, I mean, there's a lot of interesting kind of traditional ML that you could do in that space. And one of the often overlooked side effects of the boom of LLMs is we now have ML on APIs. Like, you don't have to be an ML expert to start testing with and tinkering and training and all these things.
- [40:28]
Jason Hawk
Yeah, totally. I saw a really interesting talk one time that was just like, reinforcement learning was, like, just getting going, and now everyone's been distracted by chatbots for a couple years. I'm excited for when we go back to reinforcement learning. But it's similar to your question about simulation where, like, we really want our platform to be able to support, hey, this organization is plugging into an ML vendor or an ML consultant. They've hired some whatever. Or, you know, there's a feedback loop that you need help standing up where you might have the outputs of a test and want to be feeding that back into a model model. The model's getting updated. That's going back to another team who's doing kind of like, parameter design. At some point, we are going to have to kind of like, vertically integrate there. I think that our customers tend to lead us in the direction of, like, asking for more technical expertise. But at the moment, it's like, you know, if you're training your own model, if you're bringing in something like anthropic, whatever it is, like, we want to kind of, as a platform player, be a good integrator with all of those.
- [41:21]
Kevin Ball
I guess that sort of leads to the question of, like, where do you see this going for yourself for nominal, but also just sort of this broad space of hardware development and test over the next few years and it's gotten almost cliche in software. I try not to project years anymore, but hardware is still slower. So I'm going to push you to project out a few years, not just a few months like we do in the LLM world.
- [41:44]
Jason Hawk
I think we should be projecting years in the software world too. I think we shouldn't give ourselves that pass. But there's a lot on our roadmap and a lot that I see our customers doing. But one of the things that we are working with a few different partners on right now is how to be more efficient with testing. We have, I think in a way that's very natural, develop these regimes of like intensive certification and testing that are very expensive, both from a time and a dollar standpoint. And with the advances that we've been able to achieve in machine learning and building models of things like aircraft, we can actually just do like as you said, good old fashioned data science about like, hey, how do we achieve the same level of confidence or same level of kind of statistical inference with 100 data points instead of 1,000 or 1,000 instead of 10,000. We have some partners at the Air Force who dream of a pilot who's up in the air performing a test flight, being able to get feedback from something they did five seconds ago and informing what they're going to do.
- [42:40]
Kevin Ball
Oh, that's fascinating. Right? You get to real time and you're like, hey, we saw a little bit of a flex on this. Can you try this maneuver and see what happens or something?
- [42:47]
Jason Hawk
Yeah. And if anyone knows what a knee board is, it's literally a clipboard that you have like almost duct taped to your thigh. That's like, these are the 10 things I should do while I'm up in the air experiencing G forces. Like, it's very hard to do what these guys do. It's incredible. But how do we make that an iPad and how do we make it so it's not predetermined, but dynamic? That's a dream. I think like the same kind of principles can kind of trickle down into like lots of aspects of hardware development. We see like vendor purchaser collaboration potentially happening. Right. Like, how do you reduce redundant testing or increase the fidelity of the way information is communicated about what tests were performed and what the results were. As someone who's like worked in enterprise data, like, I love this stuff. I think it's incredible. Some people maybe find it boring, but if you boil it down to like kind of the software problem, it comes down to these things like asset hierarchy and catalog and data lineage that we touched on earlier, which are just like so, so crucial to these organizations functioning well. And then it's this increasing drive to integrate the different roles in the design, prototype development, initial manufacturer tests, and then like scale manufacturing. I think a lot of these aspects of hardware development are going to be changing over the next couple of years. Like the way you lay out a manufacturing facility, it might be done by AI, maybe. Like that's an area where I know some companies are spending time. And so as those things speed up, is that shifting bottlenecks to other parts of our hardware organization or are you changing some of the paradigms? And for us, we bet that sharing a single kind of data substrate across all of that is going to make everyone move faster. A really easy example to talk about is it used to be that 100 people were focused on one test asset. And if you look at the way companies like SpaceX operate, they think in terms of how do we get it so one Engineer can monitor 100 assets, kind of inverting that, getting much more efficient. But we all know from the world of software, when you give people that level of ownership, they actually develop more understanding and they can come up with innovative changes that would otherwise be lost in bureaucracy.
- [44:49]
Kevin Ball
So far we've pretty much entirely talked about what I might talk about, like mega scale hardware, right? Like rockets and jets and all of these things. Are you working with like smaller scale stuff, drones or Internet of Things type stuff or other types of physical devices out in the world?
- [45:07]
Jason Hawk
I would say that not micro scale things. We really think that our software product can be applied in manufacturing and operational contexts where the devices themselves get smaller, but the scale, the number of them gets much larger. I would say that there are drones that we work with that are, you know, not mega scale and where a lot of the interesting part comes from, it's not that you're flying once or twice a day, it's you have hundreds of them that are flying like dozens of times a day. And that data challenge gets to be quite fun.
- [45:35]
Kevin Ball
I was going to say, how does that change the shape of the problem?
- [45:39]
Jason Hawk
It changes it a lot. Because large mega scale assets, if there's a single big problem, you might pause operations and be like, we need to figure out what happened there before we fly tomorrow versus it becomes more of the cattle than the pets, where it's like, okay, we had a thousand problems yesterday. What's the one that's the most important for us to kind of drill into? Like, when I think about the future of something like Drone delivery. It's probably, you know, it doesn't need to be 100% accurate in the way that something like whatever the new aircraft that Boeing is developing for commercial flight needs to be, but it needs to be, you know, 99.9% accurate. And that's more of an economics problem than a pure engineering problem. We talk about like fleet monitoring and fleet observability. And then it's an interesting problem because it translates from just the, not just development and test, but also to kind of like operations and maintenance. And again, it's like the logic that is being defined by the engineer who's first testing something versus trying to figure out how to ramp up production versus like, hey, this satellite is orbiting the Earth or this drone is doing dozens of deliveries a day. It does not make sense to have to re encode that logic in three different tools. That's again like part of why we're shutting Nominals. We think that can all be like frameworked away into a single platform. We are really focused on testing right now, but excited to kind of get more into operations and manufacturing in the future.
- [46:56]
Kevin Ball
Well, and that does end up smoothing your pathway a lot. Right. Instead of having to, as you get end up tool soup. Okay, it's time to operationalize everything new set of software. Got to figure it out. Well, I think we're coming close to the end of our time here. Is there anything we have not talked about yet that you think would be important to discuss before we wrap?
- [47:15]
Jason Hawk
Yeah, I think one of the things that I find exciting right now at Nominal is like, we have this organic user growth in some of our large organizations that, you know, again, when I was designing Nominal, I was not thinking about the seven person startup where the CEO was using the data. And I also wasn't thinking about environments where we don't even necessarily know the users, but they're kind of sending each other links of like, hey, I did this analysis in Nominal. Can you check it out? But we're at a kind of point in our growth where, you know, we're starting to have dozens of people log into our platform in a given week that we haven't met before. With that comes kind of like data scale challenges. And we're also kind of on the side, like I said, trying to research how AI can be incorporated into our platform. So I would say across all of that, if you're interested in picking up your software knowledge and kind of like walking across the aisle to the world of physical engineering, come talk to me.
- [48:04]
Kevin Ball
Yeah, for sure. I think The AI integration is an interesting one because one of the things enabled by LLMs is what I call like intent based UIs, right? So traditionally, if you're setting up a data dashboard, you need to understand a lot of the nuts and bolts and you need to kind of drive through and say, okay, this is where I'm getting this data and I'm putting it here and I'm putting it there and like do a lot of imperative design of your data dashboard, right? Do this, then do this, feed this to here and go here. Whereas with an LLM, if you understand broadly the shape of the data, you can just say, hey, I want something that maps a bunch of these inputs to some useful dashboards that I can watch and the AI can just do it for you. And so to your point, you can open your user spectrum to folks beyond the initial experts who know exactly how to wire A to B and say, like just those people who know what data they care about or even just what problem they care about, let the AI figure out what data is relevant for that.
- [49:00]
Jason Hawk
Yeah, if I can riff on that for a second, it's like, I'm so excited about our platform play for exactly the reasons you're saying. Where there's a lot of kind of customers, inexpensive user interfaces that we wouldn't have been able to build in the past, but now our users can build it for themselves, right? So that opens this aperture. And then I think the challenge then, and this conversation I'm having with our customers is like, well, but then when it comes to tagging, cataloging any kind of data right back, something like the events that you're trying to like canonicalize into a valuable proprietary data set, like it needs to converge. And so how do you give people the tools or give, you know, if it's agents building the software, how do you give them the tools to allow for that kind of intent based user interface? But then, I don't know, what's the right term here? The organization needs to kind of actually agree and there almost need to be laws about like, well, okay, how is the data managed in a way that's not just individual but organizationally beneficial?
- [49:49]
Kevin Ball
Yeah, absolutely. It's an interesting time, right, because with great power comes great responsibility, right? If everybody can vibe code whatever they want, then you're in trouble because you get a lot of garbage in there. But if you can use that in a way that lets you empower people, but doesn't let them screw up the underlying pieces, doesn't let them make a mess, if that makes Sense then yeah, it's really, it's an interesting world we're getting into.
- [50:17]
Jason Hawk
Yeah, I think we're early in the journey of getting more eyeballs and more data within our customers and I think that these AI technologies will just actually accelerate that and that's just good for everyone.
- [50:29]
Kevin Ball
Now really quickly on that topic, there's an interpretability challenge too. Right. Because it's very easy. You know what is the thing? Lies. Damn. Lies and statistics. Right. It's very easy to get data that tells a story that may not be true if you don't understand how it's interpreted. So I'm curious. Yeah. As you think about those user interfaces, how do you make them interpretable by the end user?
- [50:52]
Jason Hawk
I mean that's something we haven't touched on. But since the earliest days of the company, my paranoia and designing our user interface was like if you're decimating, you have a multi hour flight and you're trying to kind of plot it and allow people to zoom in on regions of interest, how do you make sure that those zoomed out plots don't tell a lie and don't hide a data point? That could be critical because it could just be a split second of behavior across a multi hour test that is really problematic and indicative of something horrible happening. Our users, they lose sleep because they're like, what if the data has a story being told in it and I just am not seeing it and I that's the thing that makes or breaks this asset or system that we're developing. So again, I think we right now at least have the benefit of like our users have a culture of being like super disciplined and as long as the tooling is not getting in the way of that, which again we prioritize heavily nominal, then they will do the right thing. I'm again not to make multi year predictions but like I am a little bit worried about like if culturally we become too dependent on automation or if the UI is being developed by an agent and there isn't that like attention to detail that we have built into the product? Like do you start to have compounding misunderstandings? Like that would be terrible. So I hope we kind of like cross those bridges in less critical realms first. You know, whatever it is, there should be some kind of outage in the software world and people will kind of like talk about how they built better risk management around it. Like let's learn those lessons in a less safety critical space and then bring them to kind of like aerospace and nuclear engineering.
- [52:24]
Kevin Ball
So you're saying you don't want your rocket designed by OpenCloud.
- [52:28]
Jason Hawk
Not yet. Not yet. Let's start with video games, and then we can move on to physical systems.