← Software Engineering Daily
Software Engineering Daily
OpenTofu with Cory O’Daniel and Malcolm Matalka
00:47:58
OpenTofu is an open-source alternative to Terraform, designed for managing infrastructure as code. It enables users to define, provision, and manage their cloud and on-premises resources using a declarative configuration language.
Loading summary
Transcript76 lines
- [00:01]
Shawn Falconer
Opentofu is an open source alternative to Terraform designed for managing infrastructure as code. It enables users to define, provision and manage their cloud and on premises resources using a declarative configuration language. OpenToFu was created to ensure an open and community driven approach to infrastructure tooling and it emphasizes compatibility and extensibility for diverse deployment scenarios. Cory O'Daniel is the CEO of MassDriver and he's a founding member of Open Tofu. Malcolm Matolka is a co founder at Terra Team and he's a founding member of Open Tofu. They join the podcast to talk about the Open Tofu project. This episode is hosted by Shawn Falconer. Check the show notes for more information on Shawn's work and where to find him.
- [01:00]
Cory O'Daniel
Corey and Malcolm, welcome to the show.
- [01:02]
Malcolm Matolka
Yeah, thanks for having us.
- [01:03]
Yeah, thank you.
- [01:04]
Cory O'Daniel
Yeah, I was super jealous of both of your setups. You got like, a lot of, like, band equipment going on. I just got like a bedroom setup. Not anybody can see this on the audio, but if you could see this, I'm definitely the person that's lacking in background decor here.
- [01:17]
Malcolm Matolka
I mean, if you want, we prepared an alternate show. We can actually do Stairway to Heaven and just like do a jam sesh if you want.
- [01:25]
Cory O'Daniel
Well, we'll see. If things start to get stale, we might have to go that direction.
- [01:29]
Malcolm Matolka
No Stairway. Denied.
- [01:32]
Cory O'Daniel
All right, well, so we're talking Open Tofu today. The project's, you know, a little over a year old. Couldn't we just start off with a bit of background on the project, like, in terms of, you know, what it is, how it got started, how both of you got involved? Maybe Corey, you want to jump that off?
- [01:46]
Malcolm Matolka
Yeah. So it was a fateful, I think, Thursday or Friday morning, I hopped on Reddit to see what was, you know, people were mad about today. And lo and behold, the license has changed for a number of Hashicorp products. And so it's funny, like, being in the open Tofu community, like, Malcolm and I are buds, despite the fact that we're competitors. I've got friends at SpaceLift, despite the fact that we're competitors. And so I immediately reach out to one of my buddies at spacelift, and I'm like, I sent him a link. I'm like, did you see this? And he's like, yeah, I saw this. We're putting together a Google Meet later today. And so we got like six, seven, eight people got on a call from all the different orgs, like, day one. And we just kind of talked through, like, what that meant for each of our businesses, what that meant for the community. And we decided, you know, looking at the landscape of iac, every single major IAC tool is owned by either a venture backed company or a public company. Right. And so like, there's none that's actually like truly open source, like owned by a foundation, not like completely being kind of run by 1org. And so we decided to put our efforts into figuring out if it was plausible to do a fork of a project as big as Terraform, and kind of went on from there. I mean, there's a lot more detail, but that was kind of the early gist of it.
- [03:01]
Yeah. And so that's early moments. So Terra Team is a pretty small company. I don't know when. When was Mass Driver founded, Corey?
- [03:07]
Oh, I think 2021, but I swear all the time that like space time doesn't work the same for me. I'm just not good with like time and dates. I think it was 2021, but I.
- [03:18]
Think we're all kind of like the same. The same age in terms of companies, but different sizes, different backing and all that. And at least from my perspective, what was really wild was everyone just jumped into working together from there. We had, I don't even want to say it was like a shared enemy in a sense, but it was more like a shared positive goal in that we thought that this should all be open and that's the right choice for people. And we all just got together, worked together, like really no drama or anything like that on our side. It just sort of happens. So like Cory and I met through that. We had like our first podcast date together shortly after that. And it's to the point where I think, you know, I sort of consider Mass Driver and Terra Team sort of sister companies. Like you have sister cities. So he's the Toledo, Ohio to our Toledo, Spain.
- [04:03]
Cory O'Daniel
Nice. I like that analogy. So you get this like, you know, Avengers of IAC together on a meat call and you all agree that there needs to be essentially like a truly open source version of this. What happens after that? Like, how do you actually go? I mean, there's a ton of companies, ton of project ideas that essentially die at the stage of that like idea. You know, it takes a lot of execution from there to actually turn it into something. So what happened next?
- [04:28]
Malcolm Matolka
Yeah, I think the thing that happened almost immediately is one org forked Terraform like pretty early. And what was wild is like, right? I mean, I think a lot of times we talk about like star counts and whatnot, and it is definitely Just like a fluffy number. But what's interesting is, like, as soon as that fork happened, like the count, people following it, like, started going up pretty significantly, right? And like, I don't know, are people starring it so they can follow the drama, or are they starring it because they're interested in the project? Right. I feel like if you're upvoting something on Reddit, you're probably there for maybe the drama, like, see what's going on. But, like starring something, like, you're not going to get like that drama feedback, right? So a lot of people say, like, people are just were jumping in early on because they wanted to, like, see what was going on. Like, we got our first piece of drama in the IAC space, right? And I don't know, to me it was like, no, these are probably people that are interested in seeing a project like this be open source, right? I mean, Boozle is not an open source license. And so then it was essentially making the manifesto. We wanted something a little bit more than just some stars. We wanted companies to actually come out and be like, yes, like, we will support this. We think this is a good idea. And, you know, we tossed our names on there kind of first because we wanted to show them that we were in on it. We kind of put, you know, what we were going to try to give to the project and we just started getting people. Like, literally the amount of people that started showing up and modifying that HTML page on GitHub, like sending pull requests, like, that was our immediate first problem, was, okay, like, this already is a lot of people to manage just for this one HTML page. Like, we do have a feat in front of us. But then it was immediately trying to figure out, like, okay, how do we go about this? How do we get a good lead for the team? Like, what does this actually look like? What's the right foundation for it? And so that was kind of the next steps was kind of taking that from just this idea that had a little bit of A Spark and GitHub Actions to, like, how do we start to get people actually involved and invested in what we're doing?
- [06:16]
Yeah, the interest was really palpable too, and not just from individuals, but other companies, some of them, because they use Terraform at that point and really felt it was the right thing to do. And others just because they want the infrastructure world to be more open, more open source, and they just want to support that. So we had support from pretty much every direction, every angle, every size, organization as well.
- [06:40]
Cory O'Daniel
So what is the role of Supporting the project like entail and what do you sort of gain from it by supporting them?
- [06:48]
Malcolm Matolka
I'll let you answer first because my answer is very different from everybody else. So I'll let you go first.
- [06:56]
All right. So the way it's organized at this point is various companies have pledged full time employees and then they hire those employees. And I think it works a little bit different for each organization. But the rough idea is you are not really a part of that company. You're sort of getting paid for it, but you're working on open tofu. So in that sense it's just giving money to this organization and everyone sort of works more with Linux foundation as a small organization. Our support is less monetary and person time and more about kind of like almost like marketing, sort of like talking about it, providing feedback. We have customers using it, so we bring that feedback to the devs on there and give them some thoughts on that and how people are using it in the real world. And what we get out of it is just a good tool, a tool that's getting better tool that actually responds to what the users are interested in. More so than necessarily what a particular organization is interested in.
- [07:53]
Yeah, I think from our side. So mine's a little spicier. Like massdriver is not a Terraform Cloud competitor, right? And so like a part of this whole change of the life license was the FAQ attached to it. And the FAQ is like it's just a laundry list of this obviously not being thought through. Like when they have to add like a changed timestamp to a single question. Like you guys didn't think this one through at all, right? And so one of the cutouts still.
- [08:18]
Updating it too, man.
- [08:19]
I mean, because people are emailing that licensing at Hashicorp and they're getting new questions ask and people are getting new bills, right? And so like for us we actually fell in this tiny little cutout in the bushel where if you're using the tool but not a direct competitor to the service, you can continue using it. And so MassDriver competes more with like their Waypoint products. We're not a direct competitor to Terraform Cloud and so we joined a little bit out of spite, like honestly. And we've done a bit of writing about this. You can find it on the MassDriver blog. When we saw this happen, like hearing that a bit about like the community is not giving back. There's a bunch of companies kind of taking from what we've built. It's like, dude, this is open source. That's the point of the thing. And I've watched videos of the Hashicorp founders talk about this exact thing. Like the importance of this being open source and being owned by the community, not by a company. And so to me, this was very obviously, I called it very early. I'm like, they're getting acquired and they're trying to capture IP before the IBM thing hit. Pure speculation at the time. And so we joined out of A, this needs to be open source, but B, like I felt honestly a little backstabbed. So like Community isn't just adding something to the repo. Community is. I've sold millions of dollars of Terraform Cloud as a consultant. I've come into companies and said this is what you need to use. That's how I've contributed. I've built modules, I've convinced teams that that is the technology to use. Like I've done my part helping build this thing over the past 10 years. And to hear, no, no, no, no, no. You haven't made a single commit to the GO repo. It's like, that wasn't cool. And so that's kind of why we got involved in it. Pure passion play for us because we're like, this was not a cool move and this does need to be open source technology, whether we compete with them or not. Like, we want to be a part of this. And honestly, I kind of saw the end of Terraform as they made that decision. And I think to date they're still putting nails in their coffin around this project.
- [10:10]
So the thing you really have to appreciate about Terraform as well, which is different from the other tools that came in through that license change is Terraform really lives and dies by the community contribution in terms of providers, modules, tooling around it. You have all this other thing, you really have to think of Terraform more like a language like Python or C. And all these different implementations exist. And C or whatever is not really valuable in that the language exists. It's valuable in that you can get all these libraries to accomplish a task or that the implementations are robust enough, you can write production software. That's really the value is. So Terraform in particular, I think feels really, really negative on the community that are providing all this other value to make it useful to everyone else in the world beyond just going to this go code base.
- [11:02]
Cory O'Daniel
Right? Yeah. And sticking with that analogy of programming languages, there hasn't really been a programming language that is a closed source programming language in a long time. Even languages like C that were Historically, closed source are now open source because essentially I think everyone recognized that a language alone within its own ecosystem only has a limited essential value because it's really about other people contributing essentially to the greater ecosystem. To sort of your point of the community, contributions have been made to, you know, Terraform historically.
- [11:33]
Malcolm Matolka
Exactly.
- [11:34]
Cory O'Daniel
So in terms of like open tofu itself, like, if I'm interested in investing my time into, you know, I want to do infrastructure as code. Like, how do I think about open tofu even outside of the sourcing, but just from like a feature perspective versus things like Terraform or cloud formation or any of the other available options on the market.
- [11:57]
Malcolm Matolka
Yeah. So I mean, as far as like investing in it, I think honestly when you look at like the state of CD report and you see the adoption of all the tools that are out there, ansible, by far is still like the most widely adopted tool. But it's, I think it's for like a different era and different problem that you're trying to solve. When you look at the things for like configuring and managing cloud resources, Terraform is by far like the leader right now. The catch with this is, like, there's still a lot of companies that haven't even adopted infrastructure as code. And that's a huge, like existential problem for all of us, right? Our software, like our data is in their software. Right. And so like thinking of like open tofu versus like some of these other tools, like, you know, you got the cloudformations of the world, you got the biceps, which is like the Azure equivalent. I think Google used to have cloud connector, cloud config, and they're doing a lot of terraform now. Like it just generates the terraform. I think that what's nice about where Terraform is is it has kind of become the lingua franca of how you manage all of these APIs at scale. Right? And the thing that kind of sucks about like the cloudformations and the biceps, like, they're great technologies if you're 100% in AWS, but almost everybody is hybrid cloud today, whether they realize it or not, right. As soon as you reach out and grab Snowflake, guess what? You're hybrid cloud. You're running in two places, right? You need to reproduce environments, you need to reproduce configs, and that's where you immediately end up in this place where as a team that may be trying to adopt infrastructure as code, if I lean into a cloud formation, I've backed my future self into a corner already and I don't even know it yet. Right. And so there's very few tools that are out there that allow you to manage any cloud easily. Right. And you can make your own provider super easy too. Right. And so you see like the Pulumi and the what's Upbounds, I always forget Upbounds Crossplane. Right. Also great tools. Also both derivatives of the original Terraform providers. Right. And so like thinking about some of these cloud specific tools versus the things that are built on the Terraform provider ecosystem, to me I'm just like any of my customers, I'm like that's a bad choice. Like it's going to back you into a corner. Like you need to use Pulumi or Terraform or Open Tofu or Upbounce thing. I already forgot the name again. And I think that's key for just like having a team that's like nimble and agile and can like support all the things they need to. Now why Open Tofu over the rest is again like I'm just saying this is a hypothetical. Like you don't know when an organization is going to be forced to change a license. And this keeps happening again and again and again and again and again in our industry. Like I'm honestly over it. Right. Like happened with Mongo, happened with Terraform. Those redis kerfuffle like it just keeps happening and it blows up in the faces of operations people. DevOps platform engineers, right? We don't have enough time, we don't have enough budget and then these license things explode in our face. And so the idea of something that is built on this provider ecosystem that can work with any cloud and can be extended to support any cloud that is not owned by anybody besides the foundation and a technical oversight committee and like a governance body like that is the most risk averse decision I think any operations or DevOps engineer can make when they're starting to take that IAC journey.
- [14:51]
Yeah, I think you really can't stress the importance of Open Tofu being part of foundation enough. Because yes, some people cynically look at it say oh those engineers are being hired by these other providers. But the reality is it's the foundation all that work happens in and it's the foundation organizing all that. And the foundation has principles and I think like even bylaws that it has to go by. So Open Tofu can never become closed source or source available only it has to stick with its license. I think that that is just really valuable given as Corey said, the prevalence of Terraform style code and Open tofu I mean we talk a lot about people managing their cloud resources, but you can manage your GitHub configuration with it, you can manage your pagerduty configuration with it. So it really covers all of this and you can put all this in one tool and that's super valuable. And knowing you can depend on it in the future in perpetuity is super valuable too.
- [15:44]
Cory O'Daniel
Yeah. And I mean and also Corey, to your point, like by using something whether it's Terraform or open Tofu, but opentofu you have the advantage of it's not going to be company control, but you're avoiding that vendor lock in that you could with the cloud formations of the world because you might be whole in on AWS today but for whatever reason sometime in the future you got to go multi cloud or you know, you start doing workloads on the snowflake or something like that where you are end up in this hybrid system.
- [16:12]
Malcolm Matolka
Yeah.
- [16:14]
Shawn Falconer
This episode of Software Engineering Daily is brought to you by Capital One. How does Capital One stack? It starts with applied research and leveraging data to build AI models. Their engineering teams use the power of the cloud and platform standardization and automation to embed AI solutions throughout the business. Real Time data at scale enables these proprietary AI solutions to help Capital One improve the financial lives of its customers. That's technology at Capital One. Learn more about how Capital One's modern tech stack data ecosystem and application of AI ML are central to the business by visiting capitalone.comtech.
- [16:55]
Cory O'Daniel
Do you think like there could have been a world where the fork of this, you know, you open sourced it, but it just didn't take off, you know, it didn't get the love that it needed to be successful. Like, what do you think you did or happened to sort of avoid that chain of events?
- [17:11]
Malcolm Matolka
I honestly, I think people were just hungry for it. Like, I don't think it's anything we did. I think it was that there was a group of people, not a single org. I think that was the key and the fact that we're actually seeing progress, right? We got the fork in quick. We immediately had to start building up a lot of stuff, right? Like there was a lot of criticism on Reddit and LinkedIn and whatnot in the early months of like, oh, you guys are forking this, but like, where are you? It's like, well, we're dealing with all the legal fallout of forking something, right? There's not a lot of legal fallout that just hitting that fork button on GitHub until you piss off a bunch of lawyers, right? And then there's legal fallout, right? So it's like we had to. It wasn't just the boostle change. They changed the licensing for the Terraform Registry. If you're not using that Terraform user agent anymore, guess what you're in violation of, of the. If you're curl. If you curl something, you're in violation of the license for the Terraform Registry. We had to build a new one, right? So like day one, it's like, oh, if we want this thing, we have to build a registry. And I think that right there showed like the dedication of the teams that were involved. We said, okay, well before we can do this thing, which we're going to do, we're going to go build an open source registry for all the modules and providers, right? And like that wasn't us just like waving the flag of like, ah, we got our own Terraform. That was us saying we have to do this work so that this project's even valid in the first place. And I think just like people seeing that dedication I think is what got them excited about it and got people to start investing. Now. The other thing that's interesting here is a lot of the early criticism of what we're doing is like, oh, this license change only affects eight to 12 companies. And that was. I don't know if I can cuss on here, but that was a big old load of cow poopy, right? I'm personally friends with multiple people in Fang that have pinned to 1.57 because they're not sure what to do. They're technically a cloud. Hashicorp's getting bought by IBM, that makes it very, very, very complicated as to whether or not they're allowed to use it internally. Almost every single CNCF project started gutting Hashicorp stuff immediately. Right? And so like there was not just us doing this, but you were seeing kind of a lot of organizations that weren't affected by the boostle change being affected by the boostle change. Right? And I think the engineers kind of adjacent to that are also seeing as like, oh, this is bigger than what Hashicorp is saying it is. It's not eight to 12 companies. This is actually causing a ripple effect everywhere. I have a lot of feelings about that, so I'll shut up. But I think that was, it was, it was the dedication and kind of seeing these ripple effects across the industry, not just across 12 companies.
- [19:37]
Cory O'Daniel
Malcolm, do you have anything to add?
- [19:39]
Malcolm Matolka
Yeah, I think again, I totally agree. It wasn't anything that we did. It was very obvious that as soon as this happened, there was people that were interested in open source solution. And I think Hashicorp kind of just made it worse on themselves too, with the license having an FAQ that kept on changing. And so if you have any litigious company, anyone with serious value at stake, it's just, it's too much unknown there. And then you have someone saying, hey, we're a foundation, it's open, come use us, and you don't have to worry about it. That's very compelling for anyone who's just not sure what the future holds.
- [20:10]
Cory O'Daniel
Yeah. And how does the community help drive which features you get built? So you forked it, you had to work through all these details, you're getting feature to parity, but now it's its own thing.
- [20:21]
Malcolm Matolka
Right.
- [20:22]
Cory O'Daniel
So how do you continue to evolve it and how does community get factored into that?
- [20:26]
Malcolm Matolka
Yeah. So the roadmap is, I think, mainly two things. One, it is maintaining that compatibility with Terraform. Right. So our release cadence is a little behind. They'll release 110, R. 110 comes later. We have to do a lot of reverse engineering for. For anybody who doesn't believe we do reverse engineering, I'll tell you what, we'd be releasing this shit much faster if we weren't, but unfortunately we have to. Right? I know. It's just like, oh, you guys are just copying shit. It's like, dude, we'd have 110 out the same day. So it's like we got to reverse engineer the APIs to get that compatibility in there. Yeah, it's just like a little bit of thought. And then it's also, I think the more important part of the roadmap is the RFC process. And that right there, like, that is like the real roadmap of where open Tofu goes. So it's like we have this, like, support that we're going to have for Terraform for the foreseeable future, and then we have our RFC process. And the importance of that is there is stuff in Open Tofu that, as a CEO of a company that's in the infrastructure space that I don't think should be in there. But guess what? Doesn't matter what I think. It matters what the community thinks. Right. And like, that roadmap being shaped by people opening RFCs, having engagement around it, like, that's community, man. Like, if somebody opens an RFC and pitches an idea and they don't write a single line of code, I consider them a part of the community and it's extremely disappointing that Hashicorp doesn't. Right. To me, that's it. Like that's the roadmap is like that compatibility, that confidence, that risk aversion of like that it's going to work with what you have today. Cutting it over. We cut 120 customers over by changing a single word in our code base. Done. Nothing ever broke. Right. Like that guarantee right there. Plus what goes in here is what the community wants. That's the roadmap. That's what people are after.
- [22:07]
Yeah, I actually went and talked to the open Tofu devs before coming on here saying, hey, is there any future vision you want me to share or where the project is going? And the answer was just we do whatever the community wants. So if you want to be involved in that, go and vote on things and go and write rfcs. That's it.
- [22:24]
Cory O'Daniel
How do those get prioritized? Like, what is the process essentially for what you have to come up with your stack rank essentially because you can't do everything. How do you know what you're doing? How does that process work?
- [22:34]
Malcolm Matolka
So there's a few paths. One is if something exists already, anyone can vote on it through GitHub. So that way it sort of tracks. It's a one to one relationship there. There's not a lot of gaming that. And it's sort of a mix of a vote and then you kind of have to have a developer go in and be like, is this reasonable to do? Is it reasonable to do in a time period where we want to release this, that sort of thing. But really voting gets stuff to the top and at least evaluate it as soon as possible. Before that point though, if you have an idea, you can submit an RFC, which is just a GitHub issue, and that will get back and forth from someone in the community. Martin is, I've submitted two RFCs, I think, and within two days it's just full of posts from Martin, I think, just going over it and blasting out ideas because he's so full of ideas. And then that goes through a process. Once it gets more finalized, it goes into a point where it can start being voted on. But really, yeah, if you're interested in something, go vote on it, comment on it. If you submit a pull request, that is absolutely going to get something going faster because the things that get voted on are what the devs in open Tofu should directly work on. But if you have an idea and you say, hey, I did the pull request and so on, review it, that'll also get it in there fast as well.
- [23:44]
Cory O'Daniel
Okay. And then can you tell me a little bit about stateful encryption and is that a feature? Am I correct in it being a deviation from Terraform?
- [23:54]
Malcolm Matolka
Was that the first feature that went out? That was the net new one. I know there's a lot of them today.
- [23:59]
Yeah, that came in like the premiere release, right?
- [24:02]
Yeah. And that one's important too, right? Like if you're using any of our tools, right. Like a, it's important to secure, right. Like it might have a password in it or whatever. But honestly, like if you're using any, like if you're not running, you know, an open source tool yourself for managing your state and your Terraform or open tofu runs and you're using something like M0 harness, massdriver, spacelift, terra team, terra, anybody. You don't want us to know your usage details of the cloud, right? Like people are always worried about the password ness, but like the ability to sell the usage data, the information behind the scenes, like encrypting that is more than just encrypting some secrets. Like it's a really important one. That was one that sat stagnant and open for years and there was excuse after excuses to why it couldn't come to be. And it's like, well, it was pretty easy. I mean for eight companies that never contributed to anything. Like we got it out pretty quickly and it's, it's pretty resilient, right. And so it's like it wasn't as hard as it was made out to be or impossible to integrate into systems that are managing state. Like it was made out to be. Just do you want that data visible or not? I feel like was like the real question and somebody was like, yes, we want to be able to see it. Right. Like, so that was one of the first ones. It's, it's an awesome feature. It's very easy to use. It's driven by a variable so you can pass in that key for encryption. There's a lot of different mechanisms for like decrypting and like rotating keys and whatnot. So I'd say like if you're flipping to open tofu today, it's easy to just like throw in a GitHub action for a plan and like see that it's all working. Now the one thing, the one little bit of lock in that you'll get is as soon as you bite on that security. If you want to go back to Terraform, guess what? Your state's in the air in terms.
- [25:43]
Cory O'Daniel
Of the Encryption key management. Is that done through like a cloud kms?
- [25:47]
Malcolm Matolka
Yeah, there's like a whole. Jeez, I can't remember exactly how many drivers there are behind the scenes, but there are a whole host of them.
- [25:54]
Yeah, there's three major ones right now. One of them is just you present the key to open Tofu, however you got it on your own and it does it there. And then there's GCP and AWS ones as well.
- [26:07]
Cory O'Daniel
Okay. So I can bring sort of my own KMS if I want, but I can also.
- [26:12]
Malcolm Matolka
And also you can go back and forth between an encrypted state and an unencrypted state. So if you decide, you know, the key management is kind of not what you want right now, or you're just not, it's not important enough to you, you can always go back after you've played with it a little while. That's one thing that's really great about the open Tofu mentality is there's also we have migration docs and then switching out is easy enough as well. So there's never this point where you switch over to open tofu and then you're stuck with it.
- [26:37]
Cory O'Daniel
You mentioned this earlier, Corey, the Open Tofu Registry and how that was one of the first big challenges of the fork. Can you talk about the process to essentially build the registry, how you went about discovering versioning modules and have those providers within the registry?
- [26:53]
Malcolm Matolka
Yeah, so I mean, the module spec is pretty well defined. You have to go around the Terraform site a bit to figure out how it works. So under the hood it's mostly just redirects. So mostly of what you see in the Terraform Registry is just mappings back to like a git repo. Right. And so we just kind of had to like go through Terraform, figure out how that worked. But then it was like, okay, well how are we going to host this thing? Like, who's going to run it? Right. And that was kind of another concern is like we didn't want necessarily like, oh, you know, mass driver will dedicate the infrastructure and own the registry. I mean, I would have loved that as a business owner. Right. It's a great little thing to get on the bottom of the site a bunch of information to get, but at the same time, like we don't want again, any company having information about like the download rates of these modules and providers. Right. So it's important that A, that registry was open source so people can see how it works and B, that we get it hosted by somebody Besides us. And so I think that was like, that may have been like our big first partnership. I'm throwing that in air quotes with. I don't want to say the wrong name. Cloudflare. Right, yeah, sorry. I mix up them and one other company fastly all the time. But I think that was the first big partnership. Right. And it's funny to like look at it but like the amount of thought and passion for the community and like how they will perceive it goes into this and I don't. I think this is one of our big problems is we haven't done a great job of talking about that passion that we have for building that trust. Right. It would have been extremely easy for any of us to have made a just a registry and thrown some compute at it. I've got more AWS credits than I can deal with. Massdriver could have thrown credits at it. But the importance of that being in the public eye and being owned by the foundation and hosted by somebody that's not one of us was also key. Like that is a part of the decision making process and I think that's one of the things that builds this trust even though we don't, you know, make a stink about it. But I honestly think we, again, like, I think that's where we haven't done a great job is like talking about like thinking through that process and making sure that it works for everyone outside of the organizations that are involved.
- [28:50]
Yeah, definitely. I mean independence is so important for open Tofu and I don't think we do as good of a job as we should to show it's really. Yes, there's money coming from various companies to pay for these employees, but it's really independent in every other way in.
- [29:05]
Cory O'Daniel
Terms of populating and keeping the registry up to date. Did you run into any rate limit challenges with the GitHub APIs?
- [29:13]
Malcolm Matolka
I don't think so. The registry isn't. It's not something that is updated super fast or it's not like Google indexing the web all the time. It's really more of a batch process and the GitHub rate limits are actually pretty high. As long as you identify yourself, you're.
- [29:30]
Cory O'Daniel
Continuing to use HashiCorp's configuration language. Like are there considerations or future plans around enhancing or evolving HCL specifically for opentofu?
- [29:42]
Malcolm Matolka
I'm not positive. I think that would be, I mean honestly, like somebody on the opentofu team submitting prs to hcl. I could see that like if we need something or if we find a bug And I would honestly hope that in the idea of community that they would accept that. But I mean, I don't think that we're going to diverge from HCL that's maintained its license. I think, you know, changing that is a very nuclear option. So I think HCL and the providers are kind of both fall under, under that, that umbrella of like if those, if those licenses are changed, like the ripple effects there are catastrophic. So I think we can just depend on that for the foreseeable future. And I mean if we have to make changes to hcl, we'll obviously submit prs to it and hopefully that goes through.
- [30:28]
Yeah, and I think if you look at the, so if you go to the Open tofu on GitHub, you can see the project, you can see what's planned for the various releases. They're usually like one or two out. And if you look at what's limiting in the existing terraform world and now tofu, it's really not language specific, it's functionality. So a big one coming out now is dynamic providers which let you say, hey, I want AWS across all of these different regions. I don't want to make a new provider for each one. I just want to like reference it like you would a dictionary in Python or something like that. And all of that fits well within hcl. It's really about the semantics of how you interpret that HCL and then how you turn that into an actual plan and all that. So really I think most people are content with what HCL can do and just getting more functionality in there in.
- [31:19]
Cory O'Daniel
Terms of some of the, you know, deviations that you've made from Terraform. We talked about the stateful encryptions, but what are some of the other changes that you've made?
- [31:28]
Malcolm Matolka
Yeah, I think one of the things, so there's a bunch I can, I can go through a couple of them. But one of the things I think is that I want to point out first before talking about the rest of them is, is another one of these ones that's important. Right. Like, so as Malcolm mentioned, like not only are there guides for like how to get into open Tofu, but like how to get out if it's not the right thing for you. And so one of the things we introduced like fairly early on is a tofu extension with the idea being that Terraform will only touch a TF file. And so if you want to start using open tofu specific functionality, you can put it in a tofu file. Opentofu reads all the files, right? And so you can actually have your plans running side by side. And Terraform will only see the Terraform stuff. Tofu will see the entire thing, right? But that's also important for adoption. It's important for like easing that onboarding story of an OPS team that's just overwhelmed with a million things to do. Like, I can start trying out Open Tofu without just absolutely blowing up all my pipelines and having to do all this painful stuff, right? So it's just like another place where it's like, we don't want to make this hard for anybody to adopt because we know that these teams already have enough stuff on their plate. But the stateful encryption, obviously, super cool. We talked about provider defined functions, but one of the other things I think is so rad. I'm a big, big, big, big, big fan of using Terraform Open Tofu now obviously as building like operational abstractions for my engineering partners, right? Like, I like when you look at the cloud, every API is like half Ops, half dev, right? Like, I don't like teams having to conflate these things in their heads. And so I like building really abstractful. And so one of the things that landed was we have a GO and a LUA provider where you can actually just toss in a main GO or a LUA file and you can actually write Go that will now execute as a part of your Open Tofu module executing, which allows you to start doing some really interesting expressions, right? To build out logic so you can actually codify your processes in. And so I'm already seeing things like we're like, okay, based off of the types of data and the team, it'll hit an API and pull that team's cost center, right? Like something dumb, something silly, but like, who remembers their cost Center ID and remembers to enter it, right? And like being able to kind of hide that away from engineers where it's just like, you just get what you want and like your cost center stuff's going to be in there. We don't have to worry about like associating costs for like, you know, bigger projects. It's done for you. You kind of be able to get to do that because you have this like richer programming language that can sit behind the scenes, but still in this declarative model. So that was one that I like. When I saw it, I was like, oh my gosh. Like, I've been like, if you see my Terraform code, I just have local blocks that are just huge. There's just loops and maps and all this logic that I could try to build into locals and now I'm just like, oh, I can just write. I can just write some go for my complex expression. So that was one that I was stoked about. Malcolm go. I don't know what Malcolm's favorite one is, but like that was. I was dancing the day that landed.
- [34:22]
So my favorite is actually ridiculously mundane, but I like it because I think it opens up a lot more automated functionality and that is early variable evaluation. And one of the things it lets you do is say you're referencing a module. You used to have to hard code the entire path to the module in your hcl, but now you can make that a variable, possibly environment variable or environment file, which has a very simple syntax and reference that variable in the module source path. So what's that let you do? Well now you can do very easily more dependabot type things where it's really easy to automatically update these versions now. So if your organization updates a module, you can feed it through your entire organization automatically with very little logic and like very little smarts associated with it. And so I really like that one. And it's one of those ones too that was just on the Hashicorp issue tracker forever. I was like, no, no, no, no, no, no, no, no, no. And then opento was like, oh, it's done. Here it is. Have fun.
- [35:23]
Cory O'Daniel
Yeah, I mean a lot of these things, like they're fairly like they sound on the surface, fairly simple, but they make people's lives so much better. You know, you're removing sort of that death by a thousand cuts situations that you tend to run into with, you know, some of this configuration in terms of like using infrastructure as code, you know, OpenToFu or otherwise. Like how do you go about, you know, teams typically go about debugging and error handling like infrastructure changes.
- [35:51]
Malcolm Matolka
It's hard. There's not great tooling for it across the board, right. And so you know, many teams, they have a staging version and they're prod and they'll just like deploy it and like eh, but like the thing that sucks about all infrastructure is none of your configuration matters if the apps that are running on top of it don't work, right? And so you can stand something up and be like, oh, open Tofu Apply was green, I can see the thing in the cloud and it's healthy. But like until you put an app on it and verify like the, the functionality that you're trying to get to happen, like none of that matters. Right? And so there's like levels to testing, it's like, hey, is the code, you know, validate? Is the code valid? Right? Does it apply? Right. A lot of people that are newer to infrastructure and cloud sometimes think, oh, it applied, everything is good. It's like, that doesn't mean you didn't have an outage for 45 minutes while it was applying, right? And so like that stuff is just very hard to catch. And so, you know, patterns that, that we do is we like internally at Master, like how we do our own is we define use cases for each piece of infrastructure, like what it's going to do. And we use there's Terraform and open Tofu test. Now that'll like test your code. But the Terra Grunt team made Terra test, right? And it's an actual. You write all your tests and go. But you can stand up other things, you know, besides your infrastructure. So you can run it. It'll do your Terraform Apply. And so, you know, we'll do things like, hey, we made this queue and we're actually going to push data into it with terratest and then pull data off of it just to make sure that like, is the IAM right. Right? Is the IAM right? You make a queue all day long, right? But if the IAM isn't what you expect it to be, that app's gonna fail when it goes in the prod, right? So we do a lot of rigorous testing from like the developer view of that infrastructure world. It's more rigorous, but we don't have a lot of outages. And we, you know, kind of build libraries for all the different things that we want to do. So it makes it real nice and re. I think that's one of the best ways to think about it. I mean the number one cause of outages today is configuration changes. So like, maybe we should be thinking a little bit higher than just like, did it apply green, did it exit zero? And like how does it function from the intended use case? And we use those use cases as drivers for our test suite. So we'll say like, hey, like, you know, we support postgres in three versions. Internally we have a development version which is like a single zone, you know, fairly straightforward. We have development serverless. If we're spinning up something for like a preview environment and then we have like a production config that does like our multi zone and stuff like that. And our, our test for our, our production one will nuke a zone, like it'll nuke one of the instances in one of the zones to make sure that we're still able to read and write. Right. And like that right there, like the developer view of the world, like the Operational Day 2 of the World is the test that I want to assert. Not simply that did it make a postgres? Like that doesn't matter, like can I use a postgres? Does it have the resiliency that as an OPS person I'm trying to guarantee and so you have to reach outside of pretty much any IAC tool to do that.
- [38:51]
Yeah. I think you really have to appreciate one thing about ICE or infrastructure as well is that it corresponds to a physical thing in the world. One of those physical things can only exist at one point in time. And I know it's very convenient for us to think about infrastructure as code, like software, but software, we generate an artifact and we can run tests against it. And if those tests pass we can deploy it to our infrastructure and it'll act the same in those two cases. But dev infrastructure will never be and can never be the same thing as your prod infrastructure. So even if everything works great in dev, you don't really know if it works in prod. So if you are doing tests, just be aware of that and understand what you're going to get out of doing tests in dev and what places they will or won't apply to in producing. So it's just a slightly different mentality in how all of that works.
- [39:44]
Yeah, I think a really good, like I'll spare everyone the details but like a really good example of what he just said is I have a buddy who works for a big public company and they have hit the IP limits of kubernetes in aws. You can't test that. Right. Like that is a very hard thing to test in an environment. Right. So like the reality is like eh, when you get to prod, like Prod's prod baby, like it's. Prod's gonna do what prod's gonna do. Like so we can do like as good as possible to test it. But like you're never gonna get prod. Prod is the one and only source of truth, no matter what anybody tells you. Like prod is the source of truth. Like what's happening there is happening there. And like to get an exact replica of what you have requires you standing up all the services and, and getting the traffic right. Like you might have something fail because of the amount of ingress that you have. That stuff's kind of hard to test. Yeah.
- [40:35]
Cory O'Daniel
Well given that, you know the number one reason for Outage is configuration issues. And I think like something like 70% of data breaches or leakage of information is due to some sort of misconfiguration that led to over privileged access. Like, you know, and I think Corey, you mentioned this earlier that you know, most organizations still today are not using these, you know, infrastructure as code approaches. Like, why is that? Like what is the blocker to adoption of this approach given that it can help solve some of these or help reduce the risk of some of these challenges for organizations.
- [41:07]
Malcolm Matolka
I want to make sure Malcolm has time to answer this because I think that we have, I think we have very different user bases. I think we both have very interesting opinions here. So you know, like mass driver, like kind of where we sit in the space. Like we have a lot of people that use the platform that like bring their own iac, but a whole bulk of our customers do not have infrastructure as code. So I actually get to talk to these folks that are like starting to make that decision and they're like, hey, which, which IAC tool should I use? Like when I start using mass driver? The biggest thing is time and time again is how little as an industry we invest in our operations teams and our DevOps teams and our platform teams I think are getting a bit more love. But when we don't invest in those teams, any OPS person that's listening to this, you are the force multiplier in your business. It's just waiting to happen. And when you invest in those teams, like that's where you get that 10x engineer, right? Like if you think about software and infrastructure as, you know, a manufacturing line and the products at the end, like Ops and DevOps is at the beginning. And if that's shitty and not working well, the whole thing's wrecked. And if that's performance tuned, the world is amazing. But when we don't invest in these teams, they don't have the time to start looking at tools. They're literally just dealing with fires all day. Right? Like I look at DevOps, I know Malcolm's there too because I see him on like the same threads. But like our DevOps, our Terraform, like our kubernetes, like you see these people like that are like discussing like what's going on in their business and like it's harrowing to read. It's like, oh my God, like you forget how good you have it sometimes when you're at Org with like well established DevOps principles. But many of these companies just like their ops teams are just underwater. They don't have the time. And so it's like, hey, why aren't you guys doing infrastructure as code? It's like, dude, I work 60 hours a week. I hate my job. Like, I can't work another 80 hours a week. But that's the thing that sucks is if you put in that little bit of effort into the automation and reproducibility, like, it pays dividends, right? So I think it's all about just finding the time. And I think that one thing that we've been brutally bad at as an industry and from The OPS to DevOps transition is, is marketing what we do within a business. I remember one of my first startups, it was a video streaming platform in 2008. Imagine that, 2008 video streaming platform in the browser. And our CEO walked by one day and was like, I don't understand what you do here. And I was like, you know how this thing has never been down? That's what I do, right? And like, that was the moment that I realized I was like, I work my ass off. Like, we had, I Remember very early 2000, we had 5,000 people watching a video stream of the Mythbusters, like on our platform. And the things stayed up and I was just like, I didn't realize that we were that good. I thought this thing was going to fall apart. And this person that was literally two seats away from me was like, I don't. I don't know what you do. And I never talked about it. Right. And I think that that on the teams, like we are these people that maybe might be a little less social. We're definitely less boastful, especially for front end engineer. But sorry, guys. I love you guys too. But. But I think, like, we don't do enough to like, show the importance and like the impact that we have in businesses. And that's why a lot of these teams kind of get loaded with all this debt. They get loaded with so much like death by and cuts all day long. They just don't get the time to like embrace that original idea of DevOps, that kaizen, that continuous improvement. And if you don't have the time to do it, you just don't have the time to do it. But, like, that's where you got to find is like, you got to find that small project, that little thing that has a big impact in your business, whatever it is that's going to take like some of the stuff off of your plate so you have the time to start investing in it. And that's what we see almost Every single time that somebody comes to mass driver and they're like, we haven't picked an IC tool yet. We haven't started doing iac. Like how do we even get started? I think the other thing for them that is also a part of this, like death by a thousand cuts is, okay, we're going to adopt IAC. Well, guess what? The business has been around for eight years and we have about 40,000 things in the cloud. So how do we even like go backwards? Right? Like that? That itself is harrowing. Right. And I think the right answer is don't just pause on that because that is a huge endeavor and it is painful. What do you have coming up next that you can start to use ISE for? That's net new.
- [45:20]
Cory O'Daniel
Great.
- [45:21]
Malcolm Matolka
Like now you can reproduce it right? Now you can understand it. Right? That's going to buy you time. Right? Let's do that again. And then when we get to the point where we got a really good established like baseline for how our business works around infrastructure as code, now let's go look at the big bag of stuff that we got to bring in. But looking at everything you have today and saying that's how we're going to start, that's a death knell. Like, you're doomed.
- [45:44]
Cory O'Daniel
Yeah, yeah. I mean, I think OPS probably needs to hire a new PR firm. But like, I mean the big challenge, I think of a lot of the like back end work that happens is it's always easier for organizations to justify like clearly customer facing applications. Right. Like if it's front end, I don't need to understand the engineering work that goes into it to see that like, okay, this is actually like, you know, touching one of my customers. But I have no idea about the magic black box that's happening behind the scenes and who does what there and why it's important and stuff like that. And it's really on, I think engineering leaders in the organization to make those, you know, business cases. But anyway, Malcolm, over to you. What are, are your thoughts on this?
- [46:24]
Malcolm Matolka
Yeah, so our user base, I would say we tend to work a lot with people who are somewhat established but deciding to transition into IAC or they're up and coming and maybe they just created an OPS team or something like that. And a lot of what we find is it's really just this cultural inertia of I'm used to clicking around the UI or we want to get to more of a structured workflow, but half of our team just doesn't want to get their credentials revoked and they'll be really upset about it if they do. So we got to ease into it. So a lot of us, what we see is just definitely cultural. And not even. Just even ones with buy in from above, just getting the rest of the team in on it. And especially in the larger organizations that decided to adopt iac, they can have groups in just other countries and on different time zones. And once you revoke those credentials and someone's really upset, you don't want to, like, deal with that in the morning, or you have a. Maybe have like a language barrier, a cultural barrier that. It's hard to explain what's going on there. So it's definitely. It's like fractal, right? Like, we're talking about the Reddit posts and you have some people coming in just saying, hey, I'm trying to do this, but I can't figure out how to do it. So if you are converting to IAC and you don't have someone who can whip up those examples really quick, when someone is, for example, trying to set up their networking layer, the person's just going to get frustrated and click around and be like, fine, it's done. I figured it in half. I did it in half a day. And their manager's like, great, that's awesome. Rather than spending three days working through how all the providers work. And it just takes a lot of time. That's why I think what Corey said about just start the new things and how you want it to be and go from there is absolutely fantastic advice for anyone doing this. It's just too big of an elephant if you look at all the things you have right now.
- [48:15]
Cory O'Daniel
Yeah, you don't have to boil the ocean. Crawl, walk, run, essentially. Well, Corey, Malcolm, this was awesome. Thanks so much for being here.
- [48:22]
Malcolm Matolka
Yeah, we appreciate it.
- [48:24]
Yeah, thank you. It was awesome.
- [48:25]
Cory O'Daniel
Cheers.