← Software Engineering Daily
Software Engineering Daily
Next-Gen JavaScript Package Management with Ruy Adorno and Darcy Clarke
00:57:18
Package management sits at the foundation of modern software development, quietly powering nearly every software project in the world. Tools like npm and Yarn have long been the core of the JavaScript ecosystem, enabling developers to install, update,
Loading summary
Transcript63 lines
- [00:01]
Host/Announcer
Package management sits at the foundation of modern software development, quietly powering nearly every software project in the world. Tools like NPM and Yarn have long been the core of the JavaScript ecosystem, enabling developers to install, update and share code with ease. But as projects grow larger and the ecosystem more complex, this older infrastructure is beginning to show its limits with performance bottlenecks, dependency conflicts, and growing concerns around supply chain security. Darcy Clark and Rui Adorno are veterans of this ecosystem. Both spent years maintaining the NPM CLI and helping guide the Node JS project, where they saw firsthand the technical debt and design trade offs that define modern JavaScript tooling. Now they're building Vault, a new package manager and registry that rethinks performance, security and developer experience from the ground up. In this episode, Darcy and Rui join Josh Goldberg to discuss how Vault works, why they believe package management needs a server side reboot, what lessons they've drawn from NPM's evolution, and how features like declarative querying, self hosted registries and real time security scanning could reshape how developers build and share JavaScript in the years ahead. This episode is hosted by Josh Goldberg, an independent full time open source developer. Josh works on projects in the TypeScript ecosystem, most notably TypeScript Eslint Other, a powerful static analysis tool set for JavaScript and TypeScript. He is also the author of the O'Reilly Learning TypeScript Book, a Microsoft MVP for Developer technologies, and a co founder of SquiggleConf, a conference for excellent web developer tooling. Find Josh on bluesky, Fostodon and dot com as Joshua Kgoldberg.
- [02:04]
Josh Goldberg
Darcy Clark and Varya Dorner the Volt Company welcome to Software Engineering Daily. How's it going?
- [02:09]
Darcy Clark
Good. Thanks for having us Josh.
- [02:11]
Rui Adorno
Yeah, thank you.
- [02:12]
Josh Goldberg
Oh I'm very excited. Just to start off, let's go in order alphabetically. Your first name? Darcy. Who are you and how did you come to work with Volt and package managers?
- [02:21]
Darcy Clark
So who I am I'm a software engineer who has been developing, I would say in JavaScript for at least about 20 years now and I got into package management by jumping headfirst into the NPM Inc. The NPM company in 2019 was hired and then shortly after had brought on Roy and worked with him very closely. We were actually a part of the NPM or the acquisition of npm by GitHub in 2020 and that was a pretty exciting time for us. We got to see what it's like to work at both aced, fast moving and rocket ship of a venture backed startup and then we also got to see what it was like to go into a very large enterprise company and that was also being acquired by the largest enterprise company that is Microsoft. And yeah, we got to support the world's largest package registry. And I really enjoyed the space and I really care deeply about community and open source. So yeah, just fell in love many, many years ago with JavaScript and fell in love with also building software and making and got the opportunity late into sort of the NPM company's life span to actually do it full time. And we're back at it again here with our new company Vault.
- [03:42]
Josh Goldberg
And you, Roy hello, my name is.
- [03:44]
Rui Adorno
Roy Adorno and also a software developer and very similar To Darcy, about 20 years of experience in this field and my story with the package management ecosystem is very intertwined with Darcy. I joined his team back at npm in 2019 and we lived together through the GitHub acquisition and we help foster the npm CLI and participated in the node ecosystem for over three years together in both these companies. And yeah, these days I'm a member of the technical steering committee in the Node JS project and currently serving as the vice chair there, which means I'm kind of help steering fostering the project. Most of that is kind of helping running the weekly meetings, but also try to just be helpful to the other collaborators in the project. And yeah, I do like to mention a little bit of my background. I've been in Canada for half my life here, but I'm originally from Brazil where I grew up. So yeah, that's kind of a little bit about me and my macro.
- [04:54]
Josh Goldberg
Great. Before we dive into all the interesting stuff that Volt is doing, I'd like to walk our listeners through what it means to be a package manager. So what is the delineation between the core language or framework base such as Node and the package manager on top of it? Something like npm. How does that all work together?
- [05:12]
Darcy Clark
So that's a great question. These things obviously have to have some mutual understanding about where things live in order for the one to be helpful to the other. And I think there's in many ecosystems now actually a dedicated package manager for the language or runtimes associated and Node really I think has we sort of trailblazed the path in the early aughts in terms of the relationship between NPM and Node having distributed essentially a package manager with the runtime for so long in terms of like the delineation of where the packages actually live, the runtime needs to be aware of where to be looking at these things. And the package manager needs to be aware of where it should be placing these things to be helpful. Right? And sometimes folks think that these things should be one and the same. Or that's a more common approach that Bun and Dino are starting to take in terms of being both a package manager and runtime. And especially with the approach Dino has taken leaning into sort of like DSM Spec and sort of browser standards, we are seeing a blurring of the lines between what it means to be a package manager and a runtime. But yeah, traditionally the handoff has been where's the location of my module are, where does the runtime look for modules? And how can the package manager place those modules in a consistent place? And the value add really for the package manager is consistency, security, great developer experience around ensuring you get access to updates. You understand that there's a net new version of a piece of software and that's huge in terms of the value add. Traditionally, if you're from developing 20 years ago, you remember consuming software directly from CDNs and you would have no clue when there was a new release of a piece of software and you were hot linking and there's a lot of problems with that strategy and or you were bundling the software yourself and you would be shipping legacy code all the time. And so package managers really add a lot of value in trying to keep your code evergreen and ensure that you are safer that way. And yeah, I think there's a lot of value in the space that is for package management or package tooling, but they definitely work closely with runtimes and there has to be some sort of common understanding about where these things should should live for them to actually be operationalized by the runtime.
- [07:34]
Josh Goldberg
That makes sense. I've seen online quite a lot of discourse about things like Core Pack about how much effort the core runtime should even put into allowing users to define which package manager they're using or those API hooks between the two of them, I guess. Roy, what's the current state of things and what do you see as the role of something like node integrating with hooks for something like a package manager?
- [07:56]
Rui Adorno
Yeah, it's a good question. The node runtime ships a package manager, right? There is that canonical story of NPM being the package manager for the runtime. And yeah, it does take energy from the project just managing that relationship. It's already one. And I think there are further complications in trying to manage that directly. The relationship with all the possible package managers you could possibly use at the Node runtime. There is that, but it was. The interesting part of the story is that Core Pack, the whole discussion, it lingered for a while within the Node project. And it was not too long ago, I think it was early this year, that the Node DSC finally sit down and had a vote to decide because it was a very polarizing topic subject within the project. On one side you have a group of people who really love that user experience and the capabilities that CorePak were enabling. And on the other side there were, you know, a group of people that were very concerned about some of the details of the implementation. And eventually we had a vote. I think it was early in the summer we decided that the project itself wants to move past the corepec story. So the idea here is that we're kind of doing a. We're doing it in a form in which it's still being shipped for a while. If I'm not mistaken, it's still included in note 25. We just shipped. It's the current version at the moment we're speaking. And the idea is to start announcing to the user base that, okay, this is going away, this is being deprecated, but really give users some time before flipping off that switch so that it can adapt the ecosystem as whole can prepare and adapt to new strategies to serve the other package managers. Right. And I think, I'm pretty sure Darcy can also build up a little bit more on the Core Pack side of things.
- [10:00]
Josh Goldberg
Before we dive too deep, can you just define what is corepack?
- [10:03]
Darcy Clark
So corepak was traditionally, or when it was first introduced, was actually under the name pmm, which was Package Manager Manager. Which is funny because that basically tells you what corepack is. It's a package manager manager. And so it was very much tailored to managing a subset of the ecosystem that we consider to be packages. Funny enough, when you think about package manager, what actually differentiates a package manager from another package? Is a package manager a package itself, like, is a runtime, a package. So you get into these sort of, like, circular thoughts and actually demonstrate definitions. And if you do think about package manager as a piece of software similar to any other package, then you start to see that, hey, it needs all the same kinds of, like, assurances we have with all of our traditional dependencies or traditional packages that we want our package managers to take care of. And that includes ensuring that there's, like, integrity, there's consistency for people's projects and what was happening in the ecosystem at the time and why, I think Mayel from Yarn, who Pushed forward. Pmm, which became corpack, was pushing forward was because he was seeing a lot of issues with consistency in terms of folks installing their Node modules. And so he was seeing that certain package managers, his or PNPM or NPM's were innovating in different ways and they would conflict when someone would use one package manager or another. And so the idea was to try to, at the project level, isolate this sort of use case of type of software that you're using within your project by creating a net new definition for this type of software that's being used or that is a dependency of your project, which in and of itself is the package manager. And so by creating a net new key top level key in the package JSON, the hope was that there could be this blessed understanding about what tool you're actually using to install your dependencies. He reached out to us at the time, so when back when we were at the NPM team, he definitely reached out to us and we talked a bit about the future of PMM and corepack before it landed in Node. There's a lot of feedback that actually was given to the Node project at the time, before it even landed. And today you can still use corepack if you'd like, if you still think it's a good solution for making sure that your teams are not potentially running into errors or issues with using different package managers in your projects. I think generally the hope with Corpak was that we would see tooling carve outs across projects, which I think is good and bad in some ways. I feel like it's a bit of a regulatory capture move to say, hey, let's sort of lock in your tooling ecosystem at this level, and anything that comes down the pipeline needs to be blessed into this new subset registry or subset ledger that we own and manage. And so that was a bit, at least from my perspective, was always like a bit of the problem I had with that creating a net new definition of tooling that needed to be managed. When I traditionally have looked at package manager as just one of my packages, and we already have had mechanisms in place at the package manager level to actually enforce good practices around making sure those things were not conflated with each.
- [13:32]
Josh Goldberg
Other, that's a good transition point then, because you've gone from working on a package manager or a package manager manager, you've gone to working on both the package manager and the registry itself. So we have, on the traditional side of things, NPM now also YARD and pnpm, then we have these kind of all in one bundlers or toolkits like Deno and Bun. And then separately you have vlt, SH or Vault. What are you doing differently? Or can you introduce to us what is the point of Volt in this multifaceted world?
- [14:03]
Darcy Clark
Totally. There's a massive opportunity I think here that exists for investment in infrastructure. What you would have seen in terms of innovation and package management for JavaScript in the last decade and a half was only focused on the client side behaviors. And so when YARN came out and PNPM came out around the same time, it was actually previously ied. I think you actually had Sultan on to talk about that. In the history of PNPM around 2016, these were just net new clients that still interacted with the same APIs that the npm client did. First class npm client, which is the public NPM registry. Right. And those APIs and those endpoints really haven't changed in 15 plus years. So the clients that are coming out even now, Dino and Bun, are still looking at those old APIs and still having to try to squeeze out performance gains and try to do things that are interesting purely at the client level. And so we think that there's a huge opportunity here to unlock the server side aspect of this, like to create net new endpoints to innovate and invest in the registry side of things. And so we've actually started a project called VSR which is complementary to the Vaults client. And VSR is an acronym for the Vault serverless Registry. It works kind of like a lightweight proxy to upstreams like npm, but also introduces a private package registry for you to self host or publish into. So we think this is a huge opportunity for us to innovate on those endpoints and create more modern ways of interacting with your packages. Part of the big problem that we see and have historically seen is a lot of wasted and redundant computer happen in all of our machines. So if you, Josh and me and Roy all install the same package, we all have to resolve the depancy graph at that edge, which seems really wasteful in terms of opportunities to cache and have a centralized understanding of the dependency graph. And so the two products that we have today that are out the Vault client, which is both backwards compatible with those legacy understandings of what a registry is, as well as communicates with vsr. What we're complementing to that is an understanding of an indexing and crawling of the dependency graph, which is what we're doing today behind the scenes. And so I think there's a huge opportunity for us to innovate on the tooling we can offer the ecosystem by sort of looking at there being an understanding of the resolved dependency graph out there that we can all share and essentially have like a global cache of versus having to do all this like redundant computer. And I think in the modern AI agentic wave that's coming, we're only going to see more and more of these machines doing essentially this resolution at the edge. And it feels like a huge opportunity to obviously save some folks some time and money and cycles and also secure the folks as well. So we think that's a different way of looking at the problem and I think that that's sort of the innovative part of what we're doing with fault.
- [17:17]
Host/Announcer
If you're using AI to code, ask yourself, are you building software or are you just playing prompt roulette? We know that unstructured prompting works at first, but eventually leads to AI slop and technical debt. Enter zenflow zenflow takes you from vibe coding to AI first engineering. It is the first AI orchestration layer that brings discipline to the chaos. It transforms freeform prompting into spec driven workflows and multi agent verification where agents actually cross check each other to prevent drift. You can even command a fleet of parallel agents to implement features and fix bugs simultaneously. We've seen teams accelerate delivery 2 times to 10 times. Stop gambling with prompts. Start orchestrating your AI. Turn raw speed into reliable production grade output.
- [18:06]
Sponsor/Advertisement Voice
At Zenflow Free, you're a developer who wants to innovate. Instead you're stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It's a flexible, unified platform that's built for developers by developers. MongoDB is acid compliant enterprise ready with the capabilities you need to ship AI apps fast. That's why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building@mongodb.com build se daily listeners quick.
- [18:38]
Host/Announcer
Question when things go wrong in production, do you know why? In minutes or hours. AppSignal is the application performance monitoring tool designed for developers who want clean, actionable insights without a huge observability bill. You get all the tools you need to fix issues before customers notice, like error tracking, performance monitoring, log management, and more. AppSignal works for teams of all shapes and sizes from startups and side hustles to SMEs and enterprise, and is especially great for teams that build with Ruby on rails, Elixir, Node JS and Python. Start your free 30 day trial and get 10% off a yearly plan with code SCD10 Go to www.appsignal.com sed that's www.appsignal.com SCD and use code SCD10.
- [19:32]
Josh Goldberg
So let's say that I'm an arbitrary team working on an arbitrary project and I have experienced pain that my installs are slow. Let's say I've got a lot of node modules in my dependency graph. Are you saying that one of the benefits say of using Volt with VSR as the backing registry is that you'll have much faster installs that I can use?
- [19:50]
Darcy Clark
Correct. We actually give you the registry proxy that you can run locally. So this is kind of a beautiful conceptually that you can self host a registry. NPM never did that or well it started at open source and it went close source in 2013 or 14. So unfortunately all the innovation from then on or any kind of improvements that were made to the registry code was privatized. So we think that there's a great opportunity here for us to give you a registry that can be co located with your client and let more of the power go into that infrastructure and server side API endpoints. And because you're running essentially a local instance of a registry, you get like round tripping benefits. You get a lot of other benefits of sort of having this proxy lightweight instance sitting alongside it. Traditionally the only other option folks would have had would be Verdaccio instance, if you know that project. So Verdaccio is like the only other project out there that folks really have available to them today to mimic a NPM registry. And so we are providing an alternative to that.
- [20:58]
Josh Goldberg
Roy, anything you want to add?
- [21:00]
Rui Adorno
Enough riddle? I think Darcy is very thorough in his answers and I appreciate that.
- [21:06]
Darcy Clark
What I should say is Roy has spent a lot of time in making an efficient graph resolution algorithm and we spent a lot of time on our log file format, which we imagine is going to be the exchange format that we'll use between the server and the client. And so if you look actually at our benchmarks that we've just recently published on our website, we actually are the fastest package manager that isn't named bun and we think that we can get there. To be quite honest, since we aren't even compiled and we're dealing with the cold start that all the other JavaScript based package managers face or the cold start problem that we all face by having to rely on a different runtime.
- [21:48]
Josh Goldberg
So yeah, let's talk about performance in the lock file. I guess a two parter question for either of both you1 what is a lock file for those who haven't had the joyous pleasure? And 2 how does that impact performance? Or how does the dependency graph impact performance of installs?
- [22:03]
Darcy Clark
I think this one's for you, Roy. I think you can explain this quite well.
- [22:08]
Rui Adorno
Yeah, it is a fun topic to talk about because the lock files serve multiple purposes, right? It is there to help you lock a given install while it's right there in the name in a sense of more. You want to make sure you're not being surprised installing some. You know, maybe this new package just got a new version and it's shipping malware. So one of the ways you try to lock your current install is by using a lock file like so the rest of the team, they just want to reproduce a given install, they can just be installing from that source. But it also serves other purposes like speeding up install because you already have a realize a fully realized graph how all these texts should look like at the end in the user machine in order for this project to work. So there is also that component that plays a big part on why block file exists. So it really serves multiple purposes like that. And at the end of the day it is also serving the humans that are managing these projects because they can keep track on how the dependencies are evolving. Right. And it's something you can audit after or during each install in order to make sure you're really getting the artifacts you're supposedly getting at the end. Right. So it has multiple purposes and it is a fine balancing act trying to serve all these different Personas, this difference and use of log file in a way that it works great for not everyone, but it's step of the way, right?
- [23:51]
Josh Goldberg
Yeah. I'm going to give you what sounds like a softball question, but I suspect it's not. Let's say that I package A that Requires B at 1.1 or higher and then package C that Requires B at 1.1.1 or higher. Why is it so difficult just to figure out what versions of packages A, B and C I want? Or why is dependency resolution one with versioning so difficult in the package space?
- [24:17]
Darcy Clark
So in general, I believe you just used some grammar, right? Like some non standard semver grammar, is that right?
- [24:24]
Josh Goldberg
Yeah, I gave you imprecise semver resolution specifiers for these packages.
- [24:30]
Darcy Clark
So this is something that a lot of folks like take for granted in our ecosystem. And in fact they take it quite for granted. In most packaging ecosystems there is no actual specification there's like a quasi specification or standard for ranges or essentially like groups to be associated together of software versions. So every time we say semver, we're kind of actually, we're stealing a term that actually doesn't have any basis for the ranges. Like ranges have not been codified in the SEMVER spec. Just so everybody knows. If you want to go check out the SEMVER spec, you're going to find that it only codifies what a version is. It doesn't codify ranges, it doesn't codify to group things together. Like how you do that is all based around how the package manager interprets that spec, but definitely in terms of like, how do we group, let's say A and B or two versions of C. When you get into sort of what we call like the diamond dependency problem, where you have two different disparate dependencies that both have a transitive, what we would call like a transitive dependency and you have them at two different explicit versions in some ecosystems. Let's take Java. You can only ever have one version of a library. So you have to get rid of that like conflict. What we would say is like a conflict between those packages that are these dependencies that have different transitive dependencies. And you have to choose one, right? You have to basically say, oh, I'm going to choose one. I'll take maybe the greater one. The beautiful thing about the JavaScript ecosystem is many years ago, Isaac Schluter, who created NPM and he was benevolent PDFL for a while taking care of Node, was able to figure out that, hey, we can actually get around this problem by allowing multiple versions of the same thing to exist in a project by nesting these dependencies, these architectures. And that's a beautiful way to kind of like resolve these, that problem. And I would say that's, I like to call that like the first layer dependency hell, where you have this issue of I have these two direct dependencies that then have a shared transitive dependency that we associate being the same thing but need to resolve to only one thing. And the way that you can do that, if they are truly in conflict, is to just have two of them have two versions of those things. And we would say that's like the nested strategy for a package manager will opt for. What's the easiest thing to do is like, we just don't ever try to share these two versions. Like, you don't ever try to do what we call as like deduplication of your dependency graph. But when you do want to do deduplication because there's many reasons you want to do that. You have to then come up with a grammar which is like that non standard grammar that we have for greedily or having sort of some slipperiness between your pansies that allows you to opt into versions that may not exist today but could exist in the future that allow for us to optimize for getting the best, most evergreen version we can without breaking software along the way. So that small little example I think really, and I'm not sure if I'm doing a great job at summarizing this, but it really, I think, unlocks many of the details and nuances of what it means to do package resolution and also the. The problem of dependency hell in general that we were facing back in the.
- [28:01]
Josh Goldberg
2000 and tens really sounds like it puts the hell in dependency hell. Yeah. Resolve multiple versions of packages with different strategies per package manager.
- [28:10]
Darcy Clark
It's one of those things that we all have to understand that there's this language the package managers have in their brains, this sort of parsing that happens of the specs that we put in our package JSON files and these dependency specs a lot of folks take for granted. They sort of are like, that's just a sembervalue. Well, actually, no, it's not. That's going to do something different potentially. And we spend a lot of time talking about grammar and definitions and specs and schemas in the package manager world. And I would say like, one of the things that I constantly ask other people is like, what do you think a package is? Like what is a package? And I get many different answers to that. Or what is the most minimal thing that could represent a package? Like what does that string look like? And I get many different answers to that.
- [29:04]
Josh Goldberg
So, all right, we've talked about the base of what a package manager is, what a registry is. I want to focus a little more on Vault and what's coming up next for y'.
- [29:12]
Rui Adorno
All.
- [29:12]
Josh Goldberg
For the next bit of the interview, you've re implemented the core parts of the manager and your registry and you've added in some niceties like working on the edge and you duplicating for better performance. What are the other selling features or parts of your product that you're excited about?
- [29:26]
Darcy Clark
So we are starting to roll out net new features to the client and the Registry one is obviously being safe by default has become standard. And what I mean by that is not running arbitrary scripts or install scripts has now become cool, which I'm very Happy about our package manager obviously takes care of this. By default, if you run Volt Install, we're not going to run any install scripts and we're going to print a nice message which we just shipped recently. Roy was actually the one to implement this feature. And we utilize our amazing query language, which is the other sort of core feature of the client, as well as we're putting this into the Registry to allow you to do sort of a nuanced allow list. If you do want to opt into running those install scripts and you do want to create some kind of configured, codified list of dependencies that should be running in, let's say install scripts, because there are legitimate reasons for that, mounting native binaries for your runtimes, or just generally building a package that you think is secure or you trust it because you've seen it before. And so that's a huge unlock. We call that. We've created a. Net new command in the vault client called Build, which takes one of our query selectors, which we think is super powerful compared to sort of what the other package managers have done, which is just used package names and package versions. The query language itself is like a huge unlock in terms of being able to expressively write both with relationships or with metadata, something that selects within your fancy graph. Very nuanced, nodeless, essentially. This language is very much inspired by css, which I have a soft spot in my heart for. Not everybody does, especially the tailwind folks out there. But it's honestly, I think, the coolest thing that we have. And we are seeing the benefits of using it in our core graph logic by unlocking something like this, which was super easy for us to implement. And we're super happy that we can say as well along with PNPM now and bun that we're safe by default, we're not running install scripts. So that's a recent unlock. Another recent shift that we had was just the ability to actually select in that query selector syntax projects across your whole system. So Roy also shipped this. It's called the host selector. You can literally write a selector that will query for any project that's being configured or installed with. And we can essentially do cross project. So if traditionally using monorepos, you'll know that it's totally easy to obviously select projects or lisp dependencies in your monorepo projects. But actually sharing dependencies, configuration or doing querying across, let's say many monorepos or many repos hasn't been possible before. So the selector essentially runs essentially in the global context of your, your system and can essentially go and query anything that you can query for across all the projects we can find on your system that are configured to the Vault client. And Roy, I'm not sure if you want to dive into more on that, but yeah, that's the high level of what the host selector does.
- [32:53]
Rui Adorno
There's so many directions we could go from here. We could dive a little bit more maybe on the query language itself, what it is, how it could work, or maybe the other. I think another pillar of that is having a browser based ui and I think that's also fundamental for a lot of folks out there to actually be able to visualize like when you're applying a query, what packages are you actually selecting? Because then you have a real time experience that allows to traverse and navigate your install using that query selector in a more queries, user friendly way.
- [33:28]
Josh Goldberg
What's an example use case of this that hasn't been brought up yet? Like let's say I'm managing a monorepo, what are some things I might use as query selector syntax for?
- [33:36]
Darcy Clark
So a lot of the time it's useful for auditing or observability, but also it's great for transposing configuration or setting and getting values from different, you know, package jsons. Let's say we actually introduce, you know, the PKG command into NPM when we were there. We've obviously carried it through to the Vault client, which we think is great. VAAN itself just introduced it in a recent minor version. They now have PKG as well. It's kind of like the fun JQ built right into your package manager. And so the great thing here is you can get and set and remove or unset values in your package JSON file. And utilizing query selector syntax, you can imagine you can start to share configs, you can share dependency definitions, you can share all types of things and actually get and set those across your projects very easily. So you can sort of transpose and automate, you know, the config definitions that live within package JSON across your monorepo. So there's a lot of automations that this sort of unlocks. There's a lot of, you know, things that we think even we haven't conceptualized yet. But the idea here is you can quickly go from having sort of like a template for your workspaces that may have, you know, a whole bunch of test scripts that you define and you can go and then action set, you know, a package set utilizing the queryselector syntax that go and pick up that definition and then apply it across all the workspaces. So we look at it as like being very composing. So the PKG is a great example of what we think is a great composable sub command within the client. And with the powerful query selector, which we've added to almost all, I think all of our commands now support essentially a flag called the scope. And the scope is essentially just a flag for defining the query selector that you'd like to apply that to. So you can even do publish, pack and publish. We have those traditional sub commands for creating and distributing packages. We also have install, obviously, we have version all those commands. They have this special flag called scope, which you can define as a query selector. And it means that that operation will be ran against that selector, like those dependencies that match that selector. So it's kind of like pnpm, if you love PNPM and their filtering syntax. It's kind of like the filtering syntax on steroids, which I know I really loved what I saw when Sultan brought out that filter syntax, it's just a little bit bespoke or baroque maybe I don't know that term, but, you know, it's a little bit foreign for I think a lot of developers to know what specific little dots or slashes you have to put. I really don't want to have to remember, like XPass type selectors, like CSS to me was just the perfect kind of, you know, in between, between something that's super expressive but also super powerful. And it only gets better with the more metadata that you put into your packages and your dependencies. And we can select by that, and we can select by other kinds of states for your packages. So, yeah, huge unlock for, I think workspace management is the selector syntax. And then we were trying to create more and more of those primitive kinds of operations like PKG or Run or even exec. All support this.
- [37:03]
Rui Adorno
Yeah. Expanding a little on your example, just to give a concrete example here that I find is pretty fun. Let's say you're syndrome, right? And you have over a thousand packages out there, right? Let's say you configure all of them using Vault Client, right? So now you could go ahead, let's say you're just migrated, you moved from Twitter and you're now on bluesky, and you want to update all your packages and to update that metadata everywhere. But let's say you have some, you know, some other maybe conditions you want to apply. Maybe you just want to Apply to the package where you're actually mentioning open collected for some reason, right? Like, so this language just gives you a very powerful way to express all that. So you could use a command like vote pkg and go ahead and set your social media handler to now point to blue sky instead. And you could be pointing, okay, let me use the host local selector so that I'm selecting across all the projects configured on my machine. And let's apply some other conditions like, okay, let's just select the ones that I'm pointing the funding to Open Collective. So this is the kind of like very, you know, expressive and advanced scenarios you could work. And the query language enables you to actually kind of have that workflow available now.
- [38:23]
Josh Goldberg
It's interesting that, but we've seen a similar pattern in many areas of tooling where at first the core base, one package at a time problem gets solved. How to dedupe packages, how to talk to a registry. And then of course, products grow bigger, needs are larger, more complex, and now we're solving problems for workspaces, monorepos, microservices and micro front ends. And one area that continuously always comes up is security, which I'd like to talk a little bit about with you too. Just looking at your blog post on the host context that you've been talking about, one of the example queries is the colon malware query, which sounds like a horrifying thing to have results at all in your project. So can you talk about the malware query selector, what it means to have malware detection at the registry and or CLI level?
- [39:09]
Darcy Clark
Totally. So it does seem super scary to run that query and maybe get results. Unfortunately, it's very important possible that you could have installed some piece of malware along the way and you don't realize when you've actually done it. Historically, that selector is actually a mutable selector which is updating all the time. So the way that works is actually we have partnerships with security insights providers like Socket, in our case is one of those key vendors that we're working with and we look at those folks, those partners like metadata providers that we're hoping to enrich your dependency graph with more and more metadata. So that makes that selector language even more and more powerful for you to codify heuristics into or for your teams, or in this case to find potentially vulnerable or insecure packages. So security we take like super seriously. We have default queries that we're trying to prevent malware from getting into. But you know, sometimes these security companies can be slow. NPM has historically been slow to realize that there is malware being distributed. And so it means that there's a active and passive scanning happening and you can actually opt in to the selector that we have actually called scanned. And so you can choose to refuse to even install stuff before it's essentially scanned. Right. So this is a beautiful thing where you can say, hey, I'm not going to trust any package. This heuristic to me is important that I actually want to only install install packages that have been scanned and we're providing which hasn't come out yet, but we're going to be providing specific vendor specific scanning selectors. So I want to make sure that socket was actually the one to scan it. And I'm going to wait until they have before I actually consume. So today we sort of mask the underlying vendor by sort of giving these nice again primitive selectors like malware or CVE and then we give you types of especially for CVEs, the risk type. So in the cases of sorry, CWE specifically are the types of CVEs you can actually put in there something like redos, like a regular expression denial of service. That type is like 1. I think the number is like 1333. So if you don't care about regular expression denial of service, because maybe you're a developer tool and a package has been maybe flagged for this inefficient regular expression, which you're like, okay, thank you. Snick. Not the throw shade or purple Purple company, old school purple company that is trying to obviously protect the ecosystem, but at the same time has kind of spammed our ecosystem with maybe not so efficient advisories. You can sort of contextually apply or contextually filter out that noise with these selectors. And so we look at both us providing more and more metadata is going to create, I think, a more secure ecosystem. We don't know how everybody will apply the heuristics to their own projects, whether or not they even consider some vendors to be suspect or not, if they want to opt into some or not. We want to give everybody the option to sort of choose. But there are certain things that we consider to be default. We don't want you to install malware, but it might get on your machine prior to it having been flagged. So that's just like how could that have happened? You know, you still want to be able to find it, right? Like let me easily find it if it did happen. And so we think it is still super powerful you for you to Be able to do a scan across all your configured projects, like we show in that in the blog post, for potentially malware that might have been previously installed that is now being flagged by any one of our vendors that says, hey, this is malware. So that's kind of how that selector works. And there's more nuances there for sure. Like Socket provides us with some great insights like whether or not a package has file system access, whether it wants network access. And we created some nice quick selectors there like FS and HTTP. And so you can quickly. And also environment secrets or environment access. So you can quickly say, if I do or don't trust a package to have file system access, I can essentially gate that by using CSS's good old not selector. Right. So I don't want anything that has file system access, like find me those packages. And we have the override mechanism that you could essentially gate. Or we have modifiers as well which are like our version of overrides, which can get the installation of those packages packages.
- [43:53]
Josh Goldberg
I wonder if you could adopt that system for other concerns beyond just malware. For example, there are community initiatives such as eat around swapping packages that are older and or slower for say, modern faster packages or just native functionality. Could you add like a colon? Slow or colon. Just use node for this now, you silly goose. Selector like that?
- [44:14]
Darcy Clark
Yes. I think there's like a couple things here where we've also flagged like the type of package. So like if it's CGS esm, like we have that insight, there are definitely options for us to help with creating variants or distributions that are easily swappable from one to another. We would love to work with the existing maintainers though, so that they can be the ones to kind of control that flow. Many years ago we actually proposed an RFC for what we call distributions, which would allow for you to. To have sort of conditional variants of a package. So you could have that full source with all your test files and stuff that a lot of people complain about, but maybe you think is still important to have all the readme docs and everything else put right into your package. But then you can have that optimized like production version that's like compiled and has very like minimal files inside of the package as well, living alongside each other under the same scope or have some sort of definition that's respected by the package manager to essentially make these two things like well understood by the package manager. And so I think distributions and variants are like the way forward in regards to that kind of bit flipping or switching between one condition or another. So whether or not it's your environment or the configuration that would tell you the package manager, that, hey, I can opt into a polyfill because, you know, I'm on an older version of Node and I need that polyfill, or I can completely drop that and use the version that basically just stubs out the native API. Now those are options as well. And we're very close with some of the folks that maintain those legacy libraries, so we're very mindful that they also want the good things to happen. I won't name names, but certain maintainers that we know of do want folks to modernize and do want folks to get off of maybe having two or three or 400 dependencies for stuff that is now built into the runtimes, they want to help with that transition as well. So the ecosystem will, I think, evolve over time, but our hope is that we can provide again those primitives that would actually help us do innovative things, not just for getting rid of polyfills or getting rid of those types of things, but to provide innovative opportunities for like native specific conditional or natively built variants to coexist in a blessed way versus the ecosystem we have today.
- [46:46]
Rui Adorno
Yeah, and from the end user point of view, a way to integrate the query language and also be doing these replacements. It's using this graph modifier feature that we shipped a while ago into the client. So it gives you a way to actually point add a package or a group of packages and replace that definition with something else that is actually, it actually happens during the graph build phase of the of the package manager. So we do have one already out, which is equivalent to the more classic overrides and we do have more already planned to add on that graph modifier support in terms of supporting things like package extensions. I think this is how some of the other clients call it, which basically allows you to define maybe new dependencies of a give at a given package right within the graph and all sorts of crazy stuff like that. Also granting some extra power to the end user in this entire supply chain.
- [47:50]
Josh Goldberg
That all sounds lovely. What are some other interesting aspects of that you think users would be pleasantly surprised to discover?
- [47:59]
Darcy Clark
Oh my goodness. Well, we've talked a little bit about the UI which Roy mentioned before, and I think that our NPM interop story is quite good. I would say that something that you would be very happy to know if you're a dev tools author or if you're someone that likes this space, we have the Best NPM docs that exist out there. I will put put my foot down and say we have the only NPM docs that exist out there. If you go looking, you will not find very many NPM registry docs specific. I should have been a little bit more specific. The NPM CLI was well documented from our time there. I would say the NPM registry was never well documented. And so if you go to Docs VLT sh, you'll find pretty comprehensive documentation which gives you obviously and it's points that exist in our registry implementation, but it also provides a quite fairly good comprehensive documentation about what the API surface looks like for the NPM registry itself. The beautiful thing here is we actually give you an interactive docs experience as well, created by and facilitated by our friends at scalyr, if you know those folks. We basically distribute with the registry and interact interactive docs experience. So when you spin up the VSR instance, you get a slash dash slash docs endpoint and there it's a scalar experience that has mechanisms for you to actually go and interact and actually play with the APIs in real time, which I think is great. So that I think will help with folks understanding what's available to them. And over time it's only going to improve and get better. And so the hope there is that other dev tools, authors or folks that traditionally would have want to build something for the NPM Registry can now be like, hey, oh, I know exactly what the request and response should look like, I know what endpoints should exist and here Volt's helping me out and we're going to maintain as best we can backwards compatibility with the NPM registry now on going forward. So that's another key, I think highlight that should be exciting for folks sort of generally about what we're doing and what we're trying to maintain. Especially because we care very deeply about compatibility, we care deeply about bringing forward the ecosystem. So I'm not sure if Roy has any key highlights. Go ahead.
- [50:29]
Rui Adorno
Yeah, of course mine are going to be tailored a little bit more to the client side of things, but it really is the small things that I really love when I'm using it. I mean of course the speed, it's very good, but you kind of like, you know, that's kind of like for us it's more of a baseline. We need to be fast. We know people are not going to be using the tool if it's not fast enough. But the little details, like for example on the query language when you're using it from the Terminal default query command. By default it prints the same tree structure that users are used with from NPMLs. Kind of the classic reference, right? For me, like personally, that's so much more usable than having the JSON blob be the default output. But also something we added to Vault LS and Vault Query is different outputs. Not only the classic JSON output which is improved, it has all these security insights metadata added to it, but we do have also Mermaid output which I find super useful, super helpful and kind of plugs well with all, you know, plugs well with hackmd I'm a huge fan pack and D like we're using all the time, I can just, you know, copy and paste basically an output and visualize my graph in the rendered view. But it could also be an ocean where it can be a GitHub issue. You could just be porting this Mermaid output, whatever platform you want to take it to. And yeah, and I think the other thing I really love is really spawning the browser based ui, right? I'm trying to understand what's going on. I want to find some information about a specific package and just being able, you know, it's like for me and I'm a big terminal user, but I do love like you know, sometimes I just want the product to take my hand, guide me to that experience, just take me to the place, let me find the information I want. I don't want to be typing and fighting about, you know, 20 different commands to type right to get there. So for me those are kind of like top of mind the things I love the most about the client.
- [52:35]
Josh Goldberg
That's really exciting stuff. Is there anything else on the technical side or about Vault that we should cover before we wrap up for the day?
- [52:43]
Darcy Clark
I would say we are actively working on the large version of crawling your fancy graph and realizing before you do, which I think walks that tightrope between the opportunity to both get performance benefits that have never been seen before as well as security guarantees that are theoretical in a lot of other tools. I won't mention SBOMs, but I just mentioned SBOMs which are highly, highly theoretical and we could have a whole episode on that. So I think there's a huge opportunity here and we're super excited about, you know, being at the forefront of it and we have a lot we're going to be shipping and sharing very soon, both in terms of better performance, better tooling, but also more of a destination for folks to hit going forward. So look out for that, I guess very soon. And yeah, we're excited about that.
- [53:42]
Josh Goldberg
Well, in that case, I just have a couple more questions for each of you. Right. What learnings could I as a developer attain from studying and understanding Brazilian Jiu Jitsu?
- [53:53]
Rui Adorno
Good question. You may ask. I do practice it. It's being. It's kind of, you know, kind of a recent thing. So I'm definitely not a big, you know, strong Brazilian Jiu Jitsu guy. I'm just there, you know, to learn and being humbled every time. But, you know, it is a great workout. It is a great way for us developers to spend all these calories. You know, it takes a lot of your muscular strength and flexibility. So it's a very healthy workout for, for us to try and, you know, and just if you're able to get it a couple of times during the week. Right. Like just getting us out of the or desk or chair every single day long. Right. Like getting in shape.
- [54:38]
Josh Goldberg
I've been told that Brazilian Jiu Jitsu, or bjj involves a lot of strategy and kind of centering yourself. And as three people who actively work on open source projects around the Internet on GitHub, I imagine it's probably quite useful for us to get practiced at centering ourselves and regulating our emotions, even in quick response times. Darcy, the last question is for you. Can you tell us about Kurt Cobain's journal?
- [55:03]
Darcy Clark
Oh, no. Yeah. I mean, as a teenager, having been a little bit of a punk myself, I think I got as a present a mass produced copy of Kurt Cobain's journal. Having loved Nirvana and beaten a bunch of punk bands, it's probably the coolest thing, but also felt very inappropriate to own and seemed, in retrospect, completely unlike something he would have wanted to have exist. So I think we should probably all burn our copies, if you have one, and let it be. So that's my thoughts on that. Having been a 14 or 15 year old, getting that, I'm sure, as I think it was maybe a Christmas gift, is both cool and insightful and probably shaped a lot of my early garage band days. But looking back in retrospect, I should have never had access to that thing. I don't think we. I don't think anybody should have had access to that thing. But it was nice to see, you know, just how creative but also how much unfortunately is very pained. I'm also, you know, around the same time I'm sure I watched Donnie Darko for the first time, which became my favorite film, so, you know, have that on vhs. That also feels like a precious Memento of my time as a teenager up in the cold. Sudbury, Ontario. Small town, cold. Sudbury, Ontario. So yeah, Kurt Co Main's journal. Let it rest. Let's all burn our copies. Hold onto your VHS though of Donnie Darko. That's a classic.
- [56:46]
Josh Goldberg
That's a great film. Could you just quickly explain the full plot and back in contact with that film, folks.
- [56:52]
Darcy Clark
I feel like unfortunately the director's cut ruined it for a lot of folks. So if you see the original, I hope you do. There was original copy without all these added things. The plot is. There's a bunny in it. There you go. There's a bunny in it. That's all you need to know. It's a beautiful. Especially around Halloween, great thriller to watch and it's one of my favorite movies for sure. And I will not. I will not ruin any of it. You can come up to your own conclusions about what did or didn't happen throughout the movie. And that's why I think is so beautiful about it.
- [57:26]
Josh Goldberg
So with all that being said, destruction is a form of creation. It's been lovely to talk to you too about Vault and package registries and clients. I'll start with you Darcy. If folks want to learn more about you or Volt on the Internet, where would they go?
- [57:40]
Darcy Clark
So you can go check out our website of VLT SH is where the home of all of our information about our company, the clients products. And we'll put announcements there as well. We put some blog posts out there. You can also follow us on bluesky. It's VLT SH or at vlt Sh just our domain. You can also check us out on x.com used to be Twitter, still is Twitter slash or at I guess vltpkg. And you can also find our GitHub all our projects along with a couple extra projects there@GitHub.com VLTEPKG as well. For myself you can follow me again x.com or Twitter arcy just my first name which I think is pretty cool. I'm old. I'm very old. Is that. That's how I got that. And as well, I'm Bluesky for myself. It's my full name@darcyclark me. My. My personal website. So that's how you can find us.
- [58:42]
Josh Goldberg
Great. And you Roy, how about you and the work you're doing on Node?
- [58:45]
Rui Adorno
I guess the place I'm more active these days is Blue Sky. So if you want you should definitely follow me there. It's R U Y Adornu. Yeah. And I think if that's too hard to find. Just. Just. Yeah, like Darcy said, go to VLT Sh. Go to the company page that has, you know, our names in it, and there are links to social media, and then you'll be able to find me there.
- [59:12]
Josh Goldberg
Excellent. Well, for Software Engineering Daily, this has been Darcy Clark, Royadona, and Josh Goldberg. Thanks for listening, everyone. Have a great day.
- [59:19]
Rui Adorno
Cheers.