← Software Engineering Daily
Software Engineering Daily
The X-Plane Flight Simulator with Ben Supnik
00:56:26
X-Plane is a popular flight simulator developed by Laminar Research. It features a first-principles physics engine, realistic aircraft systems, and a wide variety of aircraft. We wanted to understand the engineering that goes into creating a flight sim...
Loading summary
Transcript44 lines
- [00:01]
A
X Plane is a popular flight simulator developed by Laminar Research. It features a first principles physics engine, realistic aircraft systems and a wide variety of aircraft. We wanted to understand the engineering that goes into creating a flight simulator, so we invited Ben Supnick on the show. Ben is a software engineer at Laminar and he's been working on X Plane for the past 20 years. He joins the show with Kevin Ball to talk about X Plane and his career working on the simulator. Kevin Ball, or K. Ball, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co founded and served as CTO for two companies, founded the San Diego JavaScript Meetup and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow K Ball on Twitter or LinkedIn or visit his website, Kball LLC.
- [01:06]
B
Ben, welcome to the show.
- [01:08]
C
Thank you for having me.
- [01:09]
B
Yeah, I am so excited to get to dig in. This is an area of software I haven't touched before, so I'm excited to go. But let's start with you. Can you introduce yourself a little bit and share about you and X Plane and what brought you here?
- [01:21]
C
I am a software engineer. I've been working on X plane for over two decades, which is like 130 years in software engineering years. It's. If you told me back in 2005 you'll be doing this in 2025, I would have been completely surprised. But I started working on X Plane back in X plane 8. I was actually planning to get out of software engineering. My plan was to make a total career change and become an air traffic controller. And I had been an X Plane user, an enthusiast, a kind of a modder. And I called Austin up and said, you know, I'm doing this training to become an air traffic controller. I can work part time while I'm doing this. How about one last thing before I go? I'm going to design a new Siri file format for you and get you some data and like that'll be like a sort of a capstone for all the modding I've done. And 25 years later, here we are. I actually got the air traffic control training and and while I was waiting for the FAA to call me back, there was just more to do on X Plane and more to do on X Plane. So I kind of stuck around. I started working on the rendering engine and by the time the FAA actually called me, we were knee deep in x plane 8 and making progress. And I realized I was on this Flight sim track.
- [02:40]
B
Yeah, I would love to sort of dig in with you a little bit on just even what it looks like to be on a project for 20 years, because that's a pretty unusual experience. But let's start with X plane. So how would you describe what is the core architecture of a flight simulator.
- [02:53]
C
Like this, A flight simulator. Probably from a software engineering standpoint, it looks closest to a AAA game. Like, that's kind of where our DNA comes from. But our business concerns are really, really different. So we're kind of outsiders. Like if there was a room with all the AAA game people, we'd be standing in the corner. Maybe like not one of the cool kids. Because we have the same requirements for high performance, as much graphics as we can possibly give them realism. Physically based rendering has been huge for us, like the whole games industry. But to a certain extent, games live and die based on being fun. And if you don't do that, you're toast. Most games are going to be toast anyway. If you're going to do a sequel, you might just upgrade the whole engine. Whereas we are kind of a platform because people make content for the flight simulator. People make aircraft, people modify airports. The modding community is so big for us that it's almost like we're a platform as much as we're an app. So we need to have continuity basically as much as we possibly can. Right. And that's where we differ from the games, where a game engine programmer might say, well, we've got this new technique, we'll have to modify all the art. And there's these constraints. So I get my art director in and say, can we do this? And the art director says, all right, it'll cost this much. And the producer says, well, we can afford it. And then they tell all the artists, go modify these things. Never use the alpha channel. Here's the restrictions, here's the bugs you have to work around. And that's how Triple A games get made. We can't do that. There is aircraft that people have already spent man years on that are already selling. And the stuff is what the stuff is. So we have to ask ourselves, how do we upgrade the engine and still keep this content working? So it's kind of one foot in AAA games and one foot in platforms, which is sort of a unique niche that we sit in.
- [04:42]
B
Yeah, that's really interesting. So before we dive into any of those, are there any things within the core software piece that would be not recognized by somebody coming from a gaming background?
- [04:54]
C
I think from a gaming background, it's all going to look pretty familiar. Our Physics engine is completely homegrown. Austin Meyer, our founder, benevolent dictator, the only Laminar employee for gosh, like at least 15 years, developed the physics engine for himself to simulate aircraft. And his background is actually in aeronautical engineering. So he has this deep understanding of why airplanes fly. So the physics approaches used there are very different from what you might get in a Havoc or some kind of general purpose physics solver. Those systems tend to be very generalized and they're trying to give you a bunch of very simple LEGO bricks to do some physics in your game. X Plane really deeply understands airplane things like we model airfoils, we model wind flows, we break the wings down into elements. So if you came from a gain engine, you go, this is the physics engine. And then you probably go, this is not like any physics engine I've ever seen what's going on here. And it's because it's very domain specific.
- [05:56]
B
That makes a ton of sense when you have a realism requirement there that most games don't have. It doesn't have to just feel real, it has to accurately represent what somebody is going to experience in one of these planes.
- [06:06]
C
In the case of the physics, it actually goes a step further because X Plane is not only used to simulate aircraft, and that's the lion's share of our use case is people training for real aircraft with X Plane and people playing being pilots at home. We have gamers, we have a lot of older users who used to be real world pilots and they've lost their medical, you know, if you've got cardiac problems or I say problems, you can't legally go fly a single engine plane. And these people discover that they can rebuild what they had in simulators and still get some joy out of flying. But we also have people who design new planes using X Plane. And that's unusual even for flight simulators is, you know, electric. Vtols is an area that's had a lot of growth recently because the batteries have gotten good enough. And once your engine is a simple, inexpensive small electric motor and not a complex, unreliable gasoline or diesel engine or turbine, you can put four engines on, you can do all sorts of novel designs, and X Plane can give you pretty good predictions of how that's going to fly just by inputting the design. We didn't need to know the answer ahead of time and then spit it back to you, which is the conventional simulation approach. We can actually be predictive. And that puts an even higher requirement on the physics because we have to get it right, because no one's going to tell us what the answer is.
- [07:31]
B
That's fascinating. So I'm kind of curious then, what does the format for those airplanes look like? Is it similar to what someone is going to see, for example, as they're specing things out for manufacturing, or, like, what goes into that?
- [07:43]
C
The format is proprietary to X Plane and it's pretty specific to X Plane's needs. So we have an aircraft format with just. It's really just a giant key value property table full of stuff. And we, we need airfoils and their locations. We have a separate sub file to describe the airfoil. The airfoil is described as a table because we don't predict what the airfoil will do from first principles. That requires computational fluid dynamics, and that's a whole nother ball of wax. So once your airfoil has been measured, maybe you have a wind tunnel, maybe you did cfd, maybe you just downloaded the answer. Because there's a lot of airfoils that fortunately are published. But you've said, I'm going to use this airfoil. You can tell X Plane, here's my wings. These are the airfoils. This is the shape. There's a lot of shaping you can do. They can twist, they can be tapered, they can move while you fly, they can be swept. Here's the control surfaces. This is how much of the space you have. And so all of this detailed data really describes specifically how the aircraft operates in aircraft terms. So you describe flaps, you describe ailerons, and once you have all this data, including, you know, here's your engines, here's your engine specification. X Plane can take this, and we can simulate the vehicle from the ground up. So we're going to compute how much force did this jet engine put out and where on the airframe did it do it? What was the wind coming into the propeller? How was that wind modified by the propellers? As it goes through, was it redirected? Is it going to then hit the wings after? Is it going to hit the tail? Is the wing going to redirect the airflow away from a different part? And essentially we have this collection of parts, and we try to derive from the parts what is the net effect on the whole. So there's a number of aerodynamic effects that are emergent, and we don't code them in. There's this tendency of aircraft to pull to the side because of the twisting motion of the propeller. And we didn't say this is how much it pulls. We said, what does the air do and this kind of comes out by itself.
- [09:46]
B
That's fascinating. And it's fascinating because as you highlight, you don't want to do the CFD underneath it because that's super expensive. But you are doing it. Sounds like a fairly low level simulation. And seeing those emergent effects arise out of it, how do you make that perform it? I mean you guys run all the way down to running on an iPhone or an Android. Like what do you do to make that all work?
- [10:08]
C
We run on the mobile phones, but it's 2025. The phones are really fast. Like the phones are much faster than the computers that were running X Plane when I started by a lot. Even an Android craplet like a low end Android device is still much better than what we had with X plane, say 8, which would have been a single core 2 GHz pentium or something. We don't live in the dark ages anymore. The physics actually take a relatively small portion of of the performance budget of the aircraft. And I think the main reason for that is that the actual number of stations you need to simulate the aircraft isn't that bad. If you can do 40 different simulation points on the wings, that's more than enough to capture the individual effects. But 40 is not a big number for a computer, right? Like we kind of think of it as like, oh, that would take me a long time to calculate. But we when computer programs are slow, it's usually because the data input stream is really big. There's 10 million of something. So as X Plane has matured, the physics has started taking a smaller and smaller amount of the CPU budget. And we've much more spent the Moore's Law surplus on graphics because our users have a unquenchable thirst for it to look more like the real world. There are some physics things we do now that we could not have done in the single core Pentium four days. So modern X Plane has fairly complicated interactions between the different surfaces. When the windstream is deflected by one part of the aircraft that carries through to the next. And those propagating effects are kind of a higher big order than individual blade theory. So if you went back and looked at the original iPhone version of X Plane, back on the 3Gs, I think it was, which is seemed like a pretty nice phone back in the day. But now we would see this as a small computer or even the original Pentiums. We were doing all the elements, but we were doing no propagation. And that is a computationally easier problem. But even with propagation, this is still A much more manageable problem than draw everything on the entire Earth within 100 mile view, which there's a lot of stuff there and users have been always happy to have more. So that's been where we've kind of sunk our dividends.
- [12:27]
B
That makes sense. Well, and as you highlight, that is a similarity to gaming. Right. You can never get good enough graphics, you can never do it. But with your constraint of, oh, I have to be backwards compatible to all of these old files, how do you take advantage of those Moore's Law dividends in the graphics space?
- [12:44]
C
So there's a couple things. First of all, it's a window of compatibility, and the window might be bigger or smaller depending on kind of how much stress we're under with that compatibility. I would say having good separation of concerns has helped with the compatibility. So I try to think of things in terms of what is this content modeling and how are we interpreting it. Right. If you say this model is a drawable geometry with a lighting model attached, your compatibility options are zero, because it is exact in the spec of what this model is, how it will be drawn, and so you're not allowed to change things. But if you say this model is a physical material existing in the real world with some properties on the material, now the lighting model is not part of the model. Right. And so you're allowed to make the lighting model better. So anytime we can use the real world as a reference for the content, we have the option of doing better. And anytime we've said the content's behavior is exactly specified, we're stuck. So we can do a better job of simulating the aircraft if you tell us we what the aircraft is, and if you tell us how the aircraft works, we're stuck. It better work the same because you told us the how. And in the graphics field, there's a pretty good separation between the kind of what of what's in the world and how it looks. So what we found is that in x plane 11 we moved to physically based rendering and we needed people to specify their materials in a physically based manner. And we could look at the previous non physically based manner and say this is the least bad approximation we can do. We can say this color texture looks a lot like an albedo. So that's the best mapping. And if all else fails, let's do the least bad compatibility path we can. But then going from X plane 11 to 12, where we went from it's physically based to its photometric meaning. Not only is it a set of math on the Lighting that is based on what real world optics do. But we're going to try to have all the calibrated numbers be in real units. Like we're going to specify the intensity of light in candela, we're going to specify the luminance of a glowing thing in knits. When we did that, to the extent that the previous content was still giving us physical materials, giving us albedos, we could interpret those correctly because we weren't changing those rules. So a lot of that content just worked. And we'd have to look and say, what don't we know that we wish we did? And the answer is, well, it's got a self illuminating texture and they never told us the luminance. So we're going to say, okay, for all aircraft surfaces, we're going to assume the luminance is 2000 nits for everything in the cockpit. And that's the least bad compatibility we can do because we know it's an aircraft, we know the pilot can see these things. What's a realistic value they would need for the simulator to function? And you need to see your displays during the day. 2,000 nits. It's not perfect, but it's pretty good. And in that way we can kind of keep stuff working as best we can. And every time we make a change, we have to ask ourselves those. There's one other aspect which is a lot of AAA games use a lot of artist time to make things work right. They have kind of a mission, they have a budget. And if the art director says, I want all the rainforest leaves to look like X, it might be the way they do that is they have code. They might write an algorithm that processes the art, or they might tell the artist, just go do it by hand. They can make that decision. In our case, I try to be conscious of how the dividends in computer graphics are driving up the price of making content. Because when I started as a modder, one person could make an airliner. The entire thing in about six months was half a man year to make an aircraft because there just wasn't that much you could do. The mesh came from the physics model. You couldn't attach custom meshes from an authoring program. You got about 1k of color, texture and that's it. There were no physical materials. The cockpit was a 2D drag and drop thing. It wasn't a 3D interactive cockpit you could click on. There was no code to write. You got the systems in X plane and so there just wasn't that much to do. And that's all that the computers of the day could do. And as the computers get better and our users expect more. One of the problems has been that the authors making airplanes have sort of borne the brunt of this and that the labor to go into the airplane is going up. So when we're going to add things to the feature, the other thing I think about, besides how am I going to map this content, the old stuff, is how much labor are we going to bring in? Are we doing something that's going to be automatic and just works, or are we going to ask them to go and do a tremendous amount of work and kind of erode the margin and the viability of making this content at all? So, for example, we added proper 3D rain to the windscreen in X plane 12. And we take the airflow over the aircraft from that CPU physics model, we put it in a texture and send it to the gpu. And now we can drive the particle system realistically, which is totally fun because you're sitting there on a rainy day and the raindrops are falling down the windshield, and then you crank the engine and they go. And they go right off the windshield. It's very satisfying. And it just works because the GPU has the real shape of the windshield, it has the real flow of the engine, and we can just do the simulation. But it's also a feature that's relatively cheap for the authors because the shape of the windshield in 3D was something they had already built for us. They were already building a 3D model and we're reusing it. So this is very cheap. When they do the windshield wipers, we need the path of the windshield wiper in the 2D space of the particle system, which is kind of a hard thing to get because the geometry of the windshield wiper is potentially very weird. That mechanism could be almost anything that people engineer. And every airplane is a hundreds of million or billion dollar engineering effort. So there's no reason why they wouldn't just build everything bespoke. So we looked at this and said there isn't a way for us to get this information in real time from the 3D mesh that's even remotely performant. So we bake it. We have the path of the windshield wiper in the 2D space of the particle system as a color gradient where the different grayscale levels tell you these pixels would have been affected by the wiper at this timescale. So now we're starting to make it a problem for the artist because you have to give us this special magical, exactly right formatted texture to make it easy for us at runtime. What we did do is, is we built a script into blender to produce this automatically. So you could start with your windshield wiper animation, which is something you already built. You do a very small amount of annotation. You say go bake and then blender can sit there and go chunk, chunk, chunk, chunk, chunk, move the wiper, find the pixels, fill them in and export them. So we can still try to protect our authors from having to build something by hand, but in this case it's going to be done ahead of time. But we're looking at the authoring cost for the entire feature and asking by making this possible, by making it something our users expect, are we driving up the cost of making add ons or can we give the authors tools to do this in a reasonably affordable manner? And now their airplanes are better, but they're still able to produce a product whose price point isn't really changing with hopefully a similar amount of man hours.
- [20:04]
B
It's fascinating because it's an area I hadn't thought about as much. So much of software, right? The downstream dividend of things getting more powerful is you can write software cheaper because you can skip steps, you can use scripting languages, you can do all of these different things. But in graphics the expectations have risen faster than the tooling.
- [20:24]
C
That's right. I think it's telling that in software. What have we done with Moore's Law? Like we made our own jobs easier. My father was in hardware for his whole life and he'd kind of like scowl at me and say, you kids with your virtual functions, you know, we didn't make the chips faster so you could just waste all the cycles. Right? But the truth is that was kind of the most useful thing you could do would be to make software cheaper to make or to make software more reliable. I think the dominance of managed languages is telling us something about what people need out of software. Right? And it's very expensive to have humans write bug free unmanaged code. So maybe we use Moore's Law and not do that. Capital One's tech team isn't just talking about multi agentic AI. They they already deployed one, it's called chat concierge and a simplifier in car shopping using self reflection and layered reasoning with live API checks. It doesn't just help buyers find a car they love. It helps schedule a test drive, get pre approved for financing and estimate trade in value. Advanced, intuitive and Deployed, that's how they stack. That's technology at Capital One.
- [21:29]
B
On that topic, what is the underlying software stack y' all are using for Xplane?
- [21:34]
C
X plane is coded entirely in C. It's pretty much a big monolithic pile of C. We use a smattering of library code for kind of the things you would expect. Like we've got libping so we can open pings, we've got racknet for a networking sub layer, we have things like that. But it's very much ours, almost surprisingly so. And I've had people ask why are you not using SDL2 for your cross platform layer? Like why do you do it yourself? And the answer is, well, the code is 30 years old and there was a time when the thing you're asking us why we didn't use didn't exist. So at some point I think we've erred on the side of keeping what we had, which was already in production, which is a trade off. You know, maybe our solution isn't as good as a new thing that came out as library code, but also there's often a lot of client code on top of a library, which is why it's often cheaper to rewrite the library or shim it than to go rewrite all the client code. We were open GL based for a very long time because OpenGL was the first hardware API that Austin used for rendering. There was a time when explain was software rasterized on the cpu, but by the time I got there it was open GL and OpenGL runs on multiple platforms. When we wanted to move to Vulcan and Metal to modern low level graphics APIs, we had to make a decision about whether we were basically going to just shim Vulcan and Metal under what we already had, or are we going to go rewrite a chunk of the software stack. And we chose to do the hard thing and rewrite a chunk of our own code. I think if we just wanted to say, hey, it's on Vulcan. That could have been an eight week skunk works project because the API is very narrow and you just shim under it. But all of the architectural contortions that you pick up from being a long lived OpenGL application would have been still there in X plane. We would have had a great tool in Vulkan and gotten none of the benefits because we'd be holding it wrong pretty much always. So instead I think we spent three or four years of continuous work on this Sydney, Houston I and what we got in return was a real rewrite of the rendering engine itself. Under modern principles, Vulkan lets you understand the performance cost of everything you do. Whereas in OpenGL you make a call, some things happen, some pixels appear later. That's basically the whole contract. It's very loosey goosey. And because that abstraction is not well suited to modern hardware, and you can't fault SGI, they failed to look into Crystal Ball in 1991 and predict what would happen in 2025. Like that's we'll give them a mulligan. But because of that, OpenGL doesn't have good matching between what's expensive and what you do. So realistically, what happens in an OpenGL driver is the driver follows along it's s, it tries to understand what the app is doing, it does a lot of work to reprocess the information, and at some point it has to go do a lot of work. And a lot of the times that's at the very first triangle you draw, they go, okay, now we understand the whole problem. Let's go compile some shaders, let's go make sure these objects are in the right part of vram. And all this work that you wish you'd done at load time is getting done at draw time. Our users, especially the professional ones, really want X plane to be like a metronome, rock solid 60fps. Don't stutter. Stutter free operation is a requirement from the FAA for professional SIMs. So a driver that will sometimes stop and go compile a shader, because on this particular hardware that's required because of this state in a way that you couldn't predict, it's really hard to hit that 60fps limit. But we didn't have any code in X plane to go prepare the shaders in the background, because there is no prepare shaders operation in OpenGL. You can tell it you're building the shaders, but it's going to have compile them and then wait and then do the other half later when you draw. Because in OpenGL the specification for a shader is incomplete. In OpenGL you give it the source code, but you don't tell it the render target. You don't tell it the fixed function pipeline is missing most of the information. So it goes and it does it later when it has the info. So we had to restructure X plane and not just switch to Vulkan to get that actual performance win. So we went in and we rewrote a lot of code. It took a while, but in hindsight it was worth it because we have something much newer, much better matched to the actual hardware. And putting new features on top of that has actually been very straightforward.
- [26:03]
B
I think that's an example of what happens when you have this very long term view as well. And you mentioned, I think this might actually be an interesting time to dive into, like what goes into maintaining a project for 20 years. How did you go about that three to four year migration in terms of like planning, phasing, isolating things so that you didn't break stuff that was running? Like, what did that look like?
- [26:26]
C
Well, Austin has always said that he wants our time horizon to be infinity. When we think about the efficiency of what we do, right, if we say, oh, there's this thing we can do, it's going to be really great in the first three months, but it's going to be inefficient in a ten year horizon. Like, it kills him to think that we're wasting our 10 year time to hit that three year thing. Like, he wants us to think about the long game and that means we can kind of think about a roadmap of where we want to move the code. So when it came to the Vulcan port, I think it started by trying to understand the technology. We had a pretty good idea of what we weren't happy with. Like, we could really see the pain points of what was already there. So we came up with this sort of refactoring roadmap for which pieces we would rebuild and when. And we had to look at how are we going to do this in pieces? How are we going to never have the app so thoroughly broken that we can't test? Because at some point if you do that, you end up breaking a bunch of things, never noticing. And then it's really hard to put Humpty Dumpy back together again. So there's sort of a huge value in never breaking the universe. We also looked at whether we had to stick with OpenGL. One of the sort of the biggest arguments we had was, do we have to keep OpenGL working when we ship Vulkan? My answer was yes. And this is what we ended up doing. We ship Vulkan as a free Update to version 11 while keeping OpenGL. And what this got us was the ability to have a very, very long beta. Like, my God, I think that beta might have been nine months, which is atypical for us. But we just needed to have Vulcan out there and find out what it was like on real machines. Because something you get from a shipping product is, you know, what's this like out in the wild, because the wild is really different than your lab machines. And in the case of Vulcan, we had no idea what we would find. And, you know, we could not have predicted what we would learn from that beta. So having the old system running at the same time meant there was no customer going, I paid $60 for this and it's brick now and I want a refund or I'm going to fight with my credit card company or God knows what. But, like, it gave us the time and the space to do the exploration. I think if we'd said, okay, X plane 11 is OpenGL, X plane 12 is Vulcan, it would have been an easier engineering problem. We would have said, hey, in this new thing, our requirements are simpler because the union of OpenGL and Vulcan is a little awkward. But on the other hand, God, it better have worked for the first time when you hit go, because you just paid for the new version, right? And there's no safety net. You can't go back to OpenGL. And in hindsight, we did not hit that. We needed that nine months of beta to understand how Vulkan performs in the wild. What happens in Vulcan when you're doing memory management and the user launches Chrome and Chrome goes and puts stuff in vram, right? Like Vulcan tells you how much video memory you have. And by the way, it might just cut in half. And you have to survive that, right? You have to write your own code to cope with that situation, preferably without exploding in a million pieces. You know, we did not have enough information in the company, we didn't have enough experience with Vulcan before we'd written Vulcan to correctly predict that. So we had to try it in the field. So knowing that, we could then say, okay, which pieces are we going to roll out? And what we found is that there were refactors we could make to our code to use OpenGL in a better way, in ways that probably didn't make a huge difference for an OpenGL driver. But we're also fitting the shape of, of Vulkan. And once we'd made those refactors and shimmed all the OpenGL code into an abstraction layer, then we could rebuild that abstraction layer for Metal on OSX and Vulkan on Linux and Windows. And then we could kind of hit the lever and see, you know, can we shift all the trains over to the new track. And since we were going to do Metal and Vulcan for Mac and Windows, we knew we would need an abstraction layer no matter what. What. So having it also handle OpenGL and OpenGL ES in the mobile market, which we thought we'd be on for a while, that was sort of the natural path and it gave us a way to do the refactor in two pieces. First, kind of making a nice API boundary where previously the OpenGL and client code was completely fused together. And once you've completely cut out that boundary, then you can do a shim and you can switch between them without recompiling the app. And that kind of makes the refactor tractable.
- [30:45]
B
Yeah, no, that technique of introducing a boundary and new abstraction layer, decoupling things and then introducing these two different ways is a. I found that to be highly effective as well. I love what you said about the sort of long term time horizon. I feel like that's something very few of us in the software industry are in companies that support that. But that's incredible.
- [31:05]
C
Yeah, I think we're very lucky to have that freedom. And it's one of the ways that we're different than games, where often a game's time horizon is we're going to do one title. And while I have not worked in the games industry, my brother does. We have people who work for us who have been in games and you sometimes see some pretty scary code because. And people write it for good reasons. Right. Like we have to ship this. We have to ship it on this date and no one's ever going to come and refactor this. So give me the big roll of duct tape and here we go. And you know, it holds together and it ships. In our case, we know we're going to pay for ripping the duct tape off and sometimes you end up using it anyway. But we have to think about what it's going to be like, you know, in a week, in a month, in a year.
- [31:45]
B
Yeah, I wish more software business models took that approach. I think it is definitely a healthier one for the code and for the engineers.
- [31:57]
A
You're a professional software engineer. Vibes won't cut it. Augment Code is the only AI assistant built for real engineering teams. It ingests your entire repo, millions of lines, tens of thousands of files. So every suggestion lands in context and keeps you in flow where other tools stall. Augment code sprints. Unlike vive coding tools, Augment Code is built for shipping to production. And you don't have to switch tooling. Keep using VSCode, JetBrains, Android Studio or even Vim. Don't hire an AI for vibes. Get the agent that knows you and your code base best. Best start Your free trial@AugmentCode.com feeling the AI anxiety from questions to job security to cybersecurity and everything in between, it's easy to feel overwhelmed with the rate of AI innovation. Enter Aeria, the enterprise AI orchestration and security platform built to boost your confidence. With Aeria, you don't have to compromise between speed, speed in innovation or security and governance. Quickly deploy AI without cutting corners on compliance. Give your teams the confidence to adopt AI with ARIA. Ready to eliminate your AI anxiety? Visit area.com to get started for free today. That's a I R I a.com you mentioned testing.
- [33:18]
B
Can I ask what your test stack looks like?
- [33:20]
C
Our test stack, we have a couple things and it's never enough. Like I would always love to have more testing. We have Catch two which runs unit tests and we have partial coverage of our lowest stack of software. You know, we've been catching up on test infrastructure because when the company was very small and X Plane was very small, testing was kind of let's give it to them users and see what they say. For a very long time, Austin was the only person working on X Plane. So test testing was he coded it, he tested a little bit and then he gave it some people and they emailed him at his personal email address if if it wasn't right. And that was, that was everything. We've built that up a bit. So we have Catch two for unit tests. We have a Python based runner of a scripting language because any sufficiently large application has a command shell in it and X plane is no exception. And the command shell, besides providing kind of strange debugging tools and things, also provides a whole bunch of stuff for testing so we can write these automated scripts and some of them will do things like we have one that opens up every aircraft with the engines off and does the procedure by invoking commands to start the aircraft, monitors the parameters of the airplanes, it starts and fails the test if something's out of whack. So if somebody makes a change and now the electrical system is broken and the battery doesn't cause her to be amperage on the bus. Test fail. So we have a suite of those. They can also take screenshots so we can get image output. And we have Daniela, our sound engineer, does a lot of web stuff too and she built this fantastic browser based diffing program so you can pull up the before and after and spot the difference regions. So we have black box testing, we have unit testing at the low level, we have some integration tests in the middle like, you know, we have a tool you can run on every airport and it will catch invalid frequency conflicts. So when you import new airport data, you can run a one time check. I would say the struggle we have is that because X Plane is a platform, you can make an almost infinite combination of aircraft. And a lot of the bugs that are hard to catch is, well, it turns out that if you have an airplane with three engines and two are electric, one is gasoline, and you have two, two clutches, then, and that particular combination is difficult to cover. And what we don't have is a very, very, very wide suite of aircraft. X Plane ships with, I want to say, about a dozen or so really carefully built handcrafted aircraft that our team does a great job on. But because they put so much work onto them, you know, I can count them on my fingers. And so the test coverage value is less. And at times we've built simple airframes just to do testing, but I think that's an area where the coverage is poor. We have two QA engineers and they both kind of supervise usage of the automated testing resources and they also do manual testing. We do have tests where there's a, you know, old school test plan, right? It's a doc with words for what they do. And in some cases you kind of need a human to go do this because automating it would be too complicated. And most importantly, like, they just sniff out bugs. Like, they're really good at that. You know, there's a, you know, we joke about, you know, well, how many milliseconds is it going to take, Danny, to find the first bug? And they, you know, they'll come in and they'll kind of shake it and get that Spidey sense that a good QA engineer can get. And that helps us kind of catch things that would fall between the cracks that the regular test infrastructure wouldn't catch.
- [36:53]
B
There are people who just seem to have a talent for breaking things and finding things are broken. It's uncanny when you, you watch them. Actually brought another thing to mind when you mentioned the many different combinations someone could do in these third party content, whether it's an aircraft or a, you know, airport or something like that. Is there like a correctness validation that makes sense in those files or things that are describable in the format but maybe not physically possible? Like, what does that even look like?
- [37:20]
C
There are semantic validations that we can do and that we do do. I would say my own work in the scenery code has in the past been pretty slim. In terms of semantic validations, mostly due to schedule pressure. And I've been trying to pay my debts off and add more validation. You know, Austin has a fair number of validations in the physics system because he doesn't want to deal with junk data coming in. But there's also like a gray area between this is clearly wrong. Like you cannot have a, a stall speed of zero. There's got to be a stall speed versus this is very silly. Like, are you sure you really mean this? And sometimes we say, well, we saw this weird bug and we looked at it and the data was crazy. So we made the validation a little more skeptical. And then we hear back that, no, no, I built an RC plane. You know, it's one foot long, so all the numbers really are small. This isn't a typo. Like, you know, this is actually how big the prop disc is because it's made of balsa wood. So there's definitely a tension between how much we can validate the model and how open ended exploiting can be to very strange stuff.
- [38:26]
B
Yeah, for sure. And if people are using it to test out things that they might build. Right. Like this can be built, but falls off out of the sky is a valid outcome.
- [38:37]
C
That's right. Yeah. And so flies well is definitely not.
- [38:40]
B
A validity parameter on the sort of fly's well side. Another thing that you kind of briefly mentioned is FAA certification. Right. So X Plane is not just for hobbyists trying things. It's not just for people manufacturing, but people are actually using this software to certify themselves to fly. What's involved in that, what has to be true at the software level to enable that?
- [39:03]
C
Surprisingly less than you'd think, but more than you think. Not in the areas you might expect. So first of all, they don't necessarily certify software, they certify devices. So we are a software reseller and we sell a professional version of X Plane to an integrator and they're going to select the computer hardware, select a version of our software, build a simulated airframe, build the controls, maybe a fiberglass enclosure, maybe they're going to buy a motion platform. If this is a high end system, they might have multiple computers networked together, running separate displays to get more visuals. And then they're going to take this entire thing to the FAA for certification. And the FAA is less picky than you would expect about the accuracy of the flight dynamics compared to the real aircraft. But they are often very picky about the operation of the simulator. So this one always cracks me up. If your flight control hardware is not working, meaning your joystick is unplugged, your simulator has to not start. It has to like log that as an error and not run. And the inspectors will come and yank the joystick and out of the USB port. And if we keep flying, they'll fail you. And I've always thought to myself, like, who's going to go in and do flight training time? And the joystick isn't working and they didn't notice, like what world do we live in? But that is a requirement, like, because otherwise it's clearly broken. So you know, we can get tested on that. So the specific regs are quite long. And my experience with the certification process is that it's a little bit ad hoc, that different inspectors have different parts of the regulations that they will check. And so we sometimes get calls from integrators saying, oh my God, we got called on this thing. It'll be like, we haven't ever seen that before but, but you know, sure enough it's there. So we try to make the most useful, sane, generally highly functional simulator. But I would say that the regulatory structure doesn't match X Plane's business model because we have a tool that's meant to simulate all airplanes. And the really traditional simulation model is we're Boeing, we built the 737, we spent about a gajillion dollars on the R and D. We're going to get, you know, a eight figure fee every time we sell one and we're going to make a level d maximally certified 737 simulator that we have made exactly like the real airplane in every way, including using the physical flight deck in that airplane. And this simulator is a one off device to simulate this one airplane. And it's going to have and God knows 7, 8, 9 figure price tag too. And the certification process is kind of aimed at that, I suspect, like here's a simulator of this thing. So what there isn't is, is a way to say here's software that's generally useful. We certify that in, in the general case it is a jack of all trades. So we're more of an arms dealer. Like we're selling shovels to the prospectors in terms of this process. And I don't mind that because dealing with the regulatory requirements is its own complicated thing. So we're happy to do the software engineering, which we're good at and let somebody else do the certification process.
- [42:16]
B
That makes sense from a device support standpoint. Does that introduce like These are custom hardware, but underneath it they're just running devices that you already know how to manage or what does that look like?
- [42:28]
C
Yeah, so we are on Mac OS X, Linux and Windows in the desktop space and Android and iOS in the mobile space. So we're running on a lot of operating systems and most of the integrations are using Windows. Occasionally we see people using Linux, but the professional integrators are usually pretty flexible about doing what we ask them. They want to succeed, they want to have a system. They get certified. They don't want to spend a lot of time screwing around. So if they're using Windows and we say, well Upgrade to Windows 11, they'll do it. If they're using Linux and we say use a different distro, they'll probably do it. We can, we can recommend hardware. So in terms of the overall machine, usually it looks a lot like an ideal machine in terms of the specs are very good because the, you know, the cost of the hardware isn't that great compared to the whole project. The operating system isn't crazy. It's hopefully not loaded down with a bunch of really crazy mods and add ons and malware or something. So it's a pretty good starting environment. We use USB hidden for our hardware input, so it's relatively straightforward to wire up just about anything to X plane in terms of joystick or control input. We also have a DLL based plugin system and we're starting to do REST and websocket based communication with the SIM as well. And we've had UDP input for years. So they have a couple of tools they can get that they can use to get custom data into the sim. But a lot of the times their joystick hardware is going to just work. Whether they manufactured it themselves and got a hidden board or they bought something from another manufacturer. We usually you can make a config file to further specify how we interact with the joystick and you can tweak a lot of the settings in our settings, but it's pretty flexible so that usually goes well for them.
- [44:18]
B
That makes sense. We've talked about a few different places where you have this interface layer that can be configurable and can let you work across the different types of types of hardware. Graphics obviously being one input device. Are there other areas where you know you are ending up handling this diverse set of devices? I know a lot of people, you know, there's a lot of standard libraries out for doing that, but you rolled your own thing. So what does that look like.
- [44:43]
C
We generally try to keep the platform specific layer as thin as possible because anytime it's platform specific, we're coding it five times and not one time. So for example, our user interface is rendered into the app in Vulcan and Metal using our own UI kit. And so we're not writing separate win 32 forms and Mac Swift UI code. And then we need something else on Linux, maybe it's GTK or GNOME or there's a lot of chaos over there. We do something else on Android, we're not doing that. We're willing to say the X Plane UI just looks like X Plane. It's not going to be platform native, but we got to write only once, so we keep that. How very thin. We don't need that much casing because four of our five operating systems look a lot like Posix. And Windows isn't that different anymore. So the platform story has gotten pretty good. I believe the HID code we're using comes from an open source library, which does give us separate HID paths on multiple platforms because how you talk to the USB driver varies, but that makes it uniform. Vulkan and Metal look different from each other and are abstracted. But Vulkan plays on Windows, Linux and Android. Metal is on all of Apple's platforms. So we get pretty good leverage from that. We have a thin set of file system wrappers, but we don't need much. You know, we have a wrapper around the memory mapping primitives to get files in. But again, you know, it can stay pretty thin and we're not doing tons and tons of file operations, so our needs are relatively straightforward. Forward Racknet abstracts the network stack and we also have a pretty thin socket wrapper. But Windows looks enough like sockets. So overall I would say the platform layer, we can keep it thin and it's been okay. We need a way to get a native window and get some mouse input off of it. But once you write that once, you know, the amount of change in that space is pretty limited. You know, the Windows code really pretty much runs the same from, you know, when I started, it was still win 32. I mean, people slag on Microsoft, but they have kept a platform running since forever. And I think people don't realize like how much they're holding up by doing that. You know, we feel it a little bit because we've been on OSX where they've changed the ABI maybe like four or five times, they've changed the UI kit several times, they've changed the language multiple times, and you have to rewrite your stack and your users aren't going, oh, I'm really excited to get the exact same features but rewritten differently. Like you don't get anything for that. So, you know, as much as we sometimes say mean things about Redmond, like, I do recognize the value of having the thing keep working so we can focus on features and not just rewrites.
- [47:21]
B
Yes, no, they definitely understand their business consumer, I think, better than just about anyone else in the space. You mentioned features. What's on the roadmap? What types of features are you all working towards?
- [47:35]
C
Hopefully by the time people hear this, but as we are recording, our newest free patch, 12.2 should be coming out. That's a patch that's very graphics intensive. We try to have the patches be in sort of a theme area. Marco, our product manager, kind of looks at things and tries to gather up the features and, you know, find things that will be good for users, but also hit a theme and limit the testing. Right. If we touch everything, we have to test everything. So the most recent one is a graphics one and there's, there's changes pretty much everywhere. I don't think I could just rattle all of them off because there's been so much, but this is the end of a pretty long, probably even a year long effort to refine the rendering and lighting stack of the SIM and really do right what we kind of prototyped in X plane 12.0. X plane 12.0 was a difficult release for us because Microsoft had come into the market with Flight Sim 2020 while we were finishing the Vulcan port. And we were completely taken by surprise by this. They really stayed in stealth mode right up till their EA announcement of the new product. And it was just, how soon will you guys have the new version and how many features can you have in it? And that's, you know, that's a pressure cooker for release. So we did all new photometric lighting and clouds and quality was something that kind of got thrown on the floor to get that out there and just show that we were in the marketplace and, you know, people use it and they could see the potential, but there was also a lot of weird stuff. And so now the team has had the time to go in and really do a great job of working all the bugs out of the lighting and going back to the real world references and using new algorithms. And it looks like a whole new sim. It's really a big upgrade. You know, people who have used it in the company, you go back to the shipping Branch, you're like, what happened? Where all the, you know, now it just looks flat. And it's funny how in graphics everything looks fine until you see the new thing. And then we hedonically adapt so quick and you go back and go, we thought that was good. And I, I know there was a time when we first saw X plane 11 and went, wow, that looks so much better than we could have imagined. But then once you've seen X plane 12, you go back to 11 and go, yeah, that doesn't look right at all. Like, look at all the things we didn't do. Right. So the lighting, the color saturation, the clouds have gotten a big update. The interaction between them has been reworked. It's going to be a really nice visual update behind that, what I think could be 12.3 for us, but your mileage may vary. We have a number of features for systems and weather that are nearing completion. So better modeling of how clouds work, better interaction with real world data when we can pull it in. I believe we will finally ship the new weather radar, which is a very accurate simulation of the radars that are on aircraft so that the pilot can see storms ahead. That runs on the GPU and probes that weather data, but then simulates the returns and the occlusion so you get a very accurate view of when there's a storm cloud ahead of you and it's big enough, the pilot doesn't really know the weather behind it because the radar doesn't penetrate it. Right. And we simulate that. Or if there's a mountain in a way, you actually get a radar return off the mountain. You can't see the storm on the other side, but you do know there's a mountain. So Philip has done a fantastic job with that realism. We're going to ship that there's new features for the fmc and I think we're going to finally ship the new network synchronization layer, at least one part of it. So Pavel has reworked how we synchronize state over the network because you can do land flight, you can do X Plane plus external visuals. And for years we only synchronized the aircraft, which is the most important thing. But as X plane has kind of grown a zoo of other stuff that started to be inadequate and we get bug reports of, hey, I've got a three monitor, three machine setup. And the jetways aren't in the same positions across the monitors. They don't line up. And it's like, well, that jetway is a simulated object that can extend and go to the airplane and move. And the machines didn't know what the state was on the master machine. So we have new synchronization. So it's going to be exciting to ship that for the first time. This is one step in a roadmap for improving external visuals. And the team keeps working on features so we have releases beyond those which, you know, I think Marco will slip my throat if I talk about. But we can see what's coming. The code isn't ready yet and you know, it's in progress. But we can say this is going to be a valuable feature for our user. We've, you know, we put shovels in the ground, we're engineering it, and we'll see in what order they come out. Because it can be hard to predict how fast things will be get coded. Because in software engineering, sometimes you don't know what the hard problem was until you crash into it. You know, if you 100%, if you already fully understood the problem, like it's probably because you already coded it, in which case why are you coding it again? Like, there's just not a lot of repeat problems. When people build a house, it's not that different than the last house they built. And so the estimates can be very accurate because you can use all that experience and maybe if it's the first house with 3D printed concrete, that's going to be really hard to predict. But if it's the 6 millionth 2x4 framed house, it's a well solved problem. And in the simulator we're pretty much never solving the problem we already solved because we already have the code to do that. So it's always new. So it's kind of always a learning experience. Every new feature.
- [53:07]
B
Awesome. So we're getting close to the end of our time and this has been absolutely fascinating. Is there anything we haven't talked about yet that you think we should touch on?
- [53:16]
C
I think we've covered a lot of the landscape. I guess the other area where we've seen a lot of change besides Moore's Law, drives us to use new hardware and new technology to try to give more to our users. And I should say I am very thankful to have that problem because I started in video and when I was at avid technology in 1998, video was rapidly becoming a solved problem. And it was hard for the company to come up with new ideas that they could sell to their customers. And what happens is somebody comes into the market with a much cheaper product using the fact that technology is faster and they've got a lower cost engineering structure and you know, you get eaten alive from below, right? The bottom will come up from below and that's where you got to look. And this was after my time there. But I think to a certain extent their bacon got saved by hd. This need to like go upgrade all this equipment. It's like, okay, now there's something we can sell to our customers that they really, really need. But other than that, once you solve the problem, you're kind of in trouble. So you know, we used to pay for programs to unzip files. Like that was the thing you paid for. And now like we take it for granted. It's built in, right? That's a market that's gone. And that's great for all of us. That's great for users. But as a programmer, you kind of have to look at what you're doing and say, what's driving this? Like, will people want more? So we've been very lucky that our users want higher fidelity simulation and we can go use this hardware and do a better job. Job. The other area where we've changed is kind of new programming techniques and language innovation. So besides moving to Vulcan to use graphics hardware better, you know, we've really changed how we approach concurrency and that's also driven by hardware. Because when I started dual core machines were exotic. Like there wasn't really a point of coding for them because no one had them. Then we reached a point where, okay, people have dual core machines. Separating the loader from the simulator would be worth it, you know, that we could load in the background and not pause your flight. Now 16 cores. Sure, 24 cores, 32 cores. People have massively concurrent machines. And fortunately the language technology is growing too. So we have access to coroutines now in C, structured concurrency has become something that people understand. So you can program with language support for these multi core machines in a way that isn't the more equivalent of GOTO for concurrent programing. So as we develop new systems, we're also kind of coding them differently and we're seeing people use structured concurrency in the SIM and be able to write concurrent code from day one. That's pretty safe, hopefully. I mean, multithreaded code is always really hard, but you know, this is at least not quite as dangerous. And it's been interesting to see how our techniques can change based on not just better hardware, but better tools and kind of better community learning. And that kind of helps us to address that challenge. Awesome.
- [56:11]
B
Let's call that a wrap.