← Software Engineering Daily
Software Engineering Daily
pnpm with Zoltan Kochan
00:36:23
Traditional package management systems for JavaScript have faced several inefficiencies related to dependency storage, resolution, and project performance. pnpm is a fast, disk-efficient package manager for JavaScript and TypeScript projects,
Loading summary
Transcript46 lines
- [00:01]
A
Traditional package management systems for JavaScript have faced several inefficiencies related to dependency, storage, resolution and project performance. PNPM is a fast disk efficient package manager for JavaScript and TypeScript projects, serving as an alternative to NPM and yarn. Due to its efficiency and reliability, PNPM is increasingly popular for managing monorepos and large scale applications. Zoltan Kokan is a full stack web developer and the creator of pnpm. He joins the show with Josh Goldberg to talk about his background and package management in the web ecosystem.
- [00:40]
B
This episode is hosted by Josh Goldberg, an independent full time open source developer. Josh works on projects in the TypeScript ecosystem, most notably TypeScript ES Slint, the tooling that enables ES Slint and Prettier to run on TypeScript code. Josh is also the author of the O'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a live code streamer on Twitch. Find Josh on Bluesky, Mastodon, Twitter, Twitch, YouTube, and dot com as JoshUakGoldberg.
- [01:24]
C
Zoltan welcome to Software Engineering Daily.
- [01:27]
D
Hi.
- [01:29]
C
Hi.
- [01:29]
D
I'm happy to be here.
- [01:31]
C
As a consistent and now longtime user of pnpm, I'm excited to talk to you. We've got a lot of great stuff to talk about, like architectures and monorepo management. Before we get into all that though, how did you get into coding?
- [01:43]
D
I started coding at school, like at 2004. I didn't have a computer back then, or maybe it was even a little bit earlier. So I kind of had the whole journey from the invention of Internet till the invention of AI. I mean, the first couple of years I was coding without Internet. I got Internet only in 2008, so I was using just old school books and pirate content, I guess in Ukraine. Actually early on I was planning to be a painter. I was doing a lot of drawing. I didn't want to join the university or institute. It was for painters. But I missed like entry days. I mixed up the dates. So accidentally I failed that gym and later on I entered math faculty to become an engineer. That's a guess. Most of it. I started with Pascal at school and then I used C, and I used C at my first job.
- [02:59]
C
So you're saying there's an alternate universe where instead of creating pnpm you became an artist? A painter?
- [03:04]
D
Yeah, but I don't think I would be very successful at that. I was always painting like real stuff, you know, not some imaginative abstract stuff, which I think is more exciting for a painter.
- [03:20]
C
So you got your first job in C. What was that.
- [03:23]
D
So the first job was at an outsourced company, it's called Svan and they have a little office in my city, Uzhkolod, which is in Ukraine. And they had different projects for American companies. I started as a junior programmer there and I was only doing like unit tests, which is interesting because people didn't use like test gen development. People were just coding and I was writing the tests for them, which now seems weird, but that's, that's how it worked. And I was so happy when after a few months I started doing yield tasks that are not related to tests. So at that company I worked almost two years and then I joined just Tensor, which is a Q and a website where verified experts answer questions. And there I worked as a full stack developer on a. NET stack and after four years maybe at Just Answer, I started really to look into Node JS and frontend and that's when I found NPM.
- [04:38]
C
I have to ask, as someone who was in. NET before JavaScript, how did you find that transition going from the strongly typed strong foundation of. Net&c to the at the time even more so wild and wacky and loose world of Node js?
- [04:53]
D
So there are pros and cons. What I really loved about JavaScript that's before I used TypeScript is that it's so faster to prototype and to get the working little CLI or tool like for scripting. It's so much easier to use JavaScript because you don't need to deal with all that boilerplate with C, even through write the hello World CLI app. With C you need to declare a class, declare a function and stuff like that. Maybe it changed since then, I don't know because now there is more than 10 years have passed since then and I know that C has improved a lot. Now you can even use C on all platforms, stuff like that. But that was my experience back then. And also you don't need all that IDE which is actually really expensive to buy a Visual Studio license. You can just use it as an editor for JavaScript. But maybe with C it's also possible, but it's not so common.
- [06:04]
C
Yeah, these days you can use VS code for C, but it is not quite the primary path. Yeah, it is amazing to think of the trajectory that the. Net area and ethos has gone through. TypeScript recently announced that they are rewriting in Go, which at that time 10 years ago would have been completely unimaginable. Microsoft using a Google oriented or sponsored language for TypeScript development for anything other than Net. It's an amazing change, but okay, moving on. Node JS. So node JS typically ships with npm, which is their package manager, the Node package manager, as it occasionally is called. Why would you want to create something like pnpm, a different package manager than the defaults?
- [06:49]
D
So how it happened, actually, at JustAnswer we had a huge monorepo with about. Okay, it's not a huge monorepo by today's standards, but at that time it was like 140 components. And the way it worked, there was a simple script which in a for loop ran NPM install in each directory. And maybe they were linked together through synlinks or not, I don't remember. And we had like a Sinopia, which is the old name of Verdaccio, running in San Francisco, and we were using it from Ukraine. And this installation was terribly slow. In San Francisco it was faster, maybe 10, 15 minutes, I don't remember. But in Ukraine it was 30 minutes just to run and install in this monorepo. And also it used a lot of disk space, so like dozens of gigabytes because of all these duplications in node modules folders, like 100 times. I'm not sure how I found PNPM, but I found it and I tried it instead of NPM in this monorepo and it immediately improved speed drastically. So back then NPM was a lot slower. Not everyone remembers it because it was like in 2015, 16, it was npm3, it was before yarn came out. And even on small projects NPM was like. It was slower than the current npm, probably 10 times. Like the current NPM is super fast in comparison. But PNPM was really revolutionary at that time and I decided to look into it. And at that time PNPM was not really maintained. It was created by Rico Star Cruz. I think he's from South Asia now he lives in Australia, I think. And it was like a fork or like a clone. I don't know of another package manager IED created by Alexander Google. And the reason these package managers were created was because in NPM3, the structure of node modules has changed. If you remember, I don't know. In npm2, node modules used like a nested structure. So every dependency had its own dependencies in a subdirectory inside the node modules in the sub directory. So you could have really deep tree structure in the node modules. And on Windows this was causing constant issues because on Windows there's a limit to the file path. And also the other problem with this structure was that dependencies were duplicated many times inside nodemodules. So they wanted to fix these issues, and in NPM3, they changed the node modules layout to a flat layout, where now dependencies are hoisted to the root. And as an alternative to this approach, this IED package manager was developed by Alexander Google. And the idea was to use symlinks to avoid both nesting real directories and to avoid hoisting everything to the root. So that's the main reason why PNPM was created.
- [10:24]
C
You used a few terms there. There's an old joke among developers, a famous cartoon on the Internet, about how in increasing order of size, there's the planets, the stars, a black hole, and then node modules. So it's nice to see so much work has gone into reducing that size and complexity. What exactly does hoisting mean in this context? And how does that change or not change how node js resolves packages?
- [10:47]
D
So by hoisting we mean that every dependency in the dependency graph is moved to the top of your node modules directory. So even if you only have like webpack in your dependencies, webpack itself has like hundreds maybe of dependencies of its own. And if you install webpack with NPM and you open your node modules directory, you'll see hundreds of directories there, even though in your packages you see only one dependency, which is webpack. This is a simple explanation of hoisting. If in the dependency graph there are multiple versions of a package, then this hoisting algorithm will pick one of these versions and put it to the root of node modules, and the other ones will be nested in one of the dependencies sub node modules directory.
- [11:43]
C
Okay, and then you also mentioned symlinks. Now, as a user of pnpm, what we experience is that you can install pretty darn quickly. And oftentimes, if it's the same installation you've done on a previous run on your device, you can just do it offline or even on an airplane. But how does symlinking play into all of this? And what even is symlinking?
- [12:03]
D
I'm not sure the symlinks have a role in the speed. It just allows us to use an alternative layout for structure in node modules. Actually, you also asked about node JS resolution in the previous question, but I did not answer, so I can mention it now. So the way node js resolves dependencies is that it searches for nodemodules directories in the current directory of the js file that has the required statement, and then Also looks in every parent directory as well. And when it finds this dependency, it resolves it. And an interesting important point here is that for this resolution, it uses the real location of the module, not the SIM link location. So it means that we can have a SIM link at one place, but we can have all the dependencies at a different place where the symlink points. So with pnpm, if you run PNPM install this webpack, for instance, you'll see only webpack in the root of the node modules directory, and it will be actually a SIM link. And you'll also see a hidden directory pnpm. And if you open up the PNPM directory, you'll see lots of directories there, like in the form of package name at and version. So this is a flat directory structure, even flatter than the NPM one with the hoisted node modules, because it has the version also of the package. So the webpack in the root of node modules points to a directory in this PNPM directory. And if you open any directory inside this pnpm, you'll see another node module there and you'll see all the direct dependencies of each dependency. So if you open webpack in pnpm, you'll see symlinks to the direct dependencies of webpack. And because when you require webpack from your project node, JS will resolve the symlink to that hidden directory inside PNPM and it will actually look for dependencies, dependencies of webpack in that isolated directory inside pnpm.
- [14:38]
C
That's fascinating. Yeah, it's really, I think, a great example of doing less work to achieve just a really clean solution to a common problem. And as a result to users, it just feels seamless. Like you run PNPM install and it feels the same as an NPM installation, just faster. But under the hood, it is kind of shockingly different.
- [14:56]
D
Yeah, the speed difference is, is actually achieved by something else. It was also a part of the initial PNPM version when I found it. So if you open the PNPM website and go to, I think the motivation page, you'll see there a nice illustration of what makes PNPM so performant. So instead of having like separate installation stages, where in the first stage all the dependencies are resolved, then a separate stage when all dependencies are fetched, and the third stage, when all dependencies are written to the node modules directory in pnpm, there are like separate independent pipelines for each packages. So one of the package will be resolved, fetched, and written to node modules at the same time when others are still being resolved, and independently at different stages of the installation. Actually, even Yarn has these different installation stages. But still, it's fast enough.
- [16:03]
E
I think your team is generating more code than ever, but you're still stuck with rigid legacy tools. Inflexible workflows, manual updates and siloed communication are slowing you down, just as your engineers juggle more pull requests and context switches than ever. That's why there's Monday.com's dev platform. With fully customizable workflows you can ship faster. No admin bottlenecks, no clunky add ons. Let your developers work in Monday dev or right from their ide. With AI powered integrations that keep every task in context, get full visibility into progress, performance and risk, all in real time, fully synced with GitHub and your entire ecosystem and with business connectivity built in, Monday dev keeps engineering priorities aligned with the impact that matters most. No more admin bottlenecks. Visit money.comdev to learn more.
- [16:54]
C
Before we move on to the higher level features like workspaces, it would be great to get your opinions on the current state of package managers. As you noted, Node and NPM themselves have changed a lot and also Yarn has had quite a few versions. Do you see any particular pros cons feature comparisons between the current versions of those three package managers?
- [17:15]
D
So I believe currently if you check PNPM is most popular for using in Monorepos. I don't know actually how good are the other package managers at monorepos? I really stopped comparing at some point but people say that it currently has the best feature set for monorepos. You can see also on our website on the workspace page at the end of the page we have like usage examples I think and there are some big names there like GitHub projects with thousands of stars like Nuxt or Vite. Some well known projects use PNPM for their monorepos. There are some features that are available on the in one of the three package managers like Yarn has the Plug and Play feature which we kind of have also, but it's not really well maintained so we use Yarn's libraries for that. So if you need Plug and Play I would recommend using Yarn. BNPM has some features that others don't have. Like recently we have added Configurational dependencies which is basically a new type of dependencies that run before everything else and you can load settings from there or use it for plugins. It's really new So I think it will be very powerful, but it needs more work to be done. But I think Yarn also has some plugin system. I don't know how it works, but I think they have like core plugins which are installed by default and they have maybe some community plugins. In pnpm we have hooks like in pnpm file CJS you can hook into different parts of pnpm.
- [19:10]
C
It's always lovely talking to open source maintainers. Very rarely does anyone who's not an open source maintainer in one answer mention things like we use one of our direct equivalent packages or projects as a dependency, such as you do with Yarn, or for this particular feature, although we have it, you should go use our other equivalent project for certain scenarios. It's really lovely to see that kind of cooperation in the ecosystem.
- [19:35]
D
Yeah, well, I've worked on pnpm for like 9 years maybe. So many things have changed since then. But Yarn's core contributor was Mael. He has also worked on Yarn for six years maybe or more. So we have talked a lot and we had, you know, when there is some hate in the community, you can empathize with that. Like I wouldn't talk about Yarn. I know how much work goes there.
- [20:08]
C
Yeah, there is an unfortunate amount of community virtual. But you're all working towards a better future and it's really lovely to see. And speaking of that future, let's talk about some of the features of PNPM that you mentioned. So just to take what you said about hooks or the configurational dependencies as an example. Why would I want, let's say, a configurational dependency if I'm managing my team's package or projects?
- [20:33]
D
Yeah, I think I was a little bit inspired with bit for configurational dependencies. I work for bit for like three years, I think, or more three, four years. If you haven't heard about bit, it's like a GitHub for component based development. It has its own version control system, package manager, like hosting, CI, everything to make component based development easier. What makes it interesting is that you don't have to choose between a mono repo and single package repo because you have like dynamic workspaces where you can fetch only those components which you need for a particular feature. So it solves the scaling issue of monorepo, but has all the pros of the monorepo. And one of the ways that BIT makes it possible is that it has these environments for components and the environment has all the necessary configuration, like not only the dev dependencies, but also Any rules for the linter for the bundler? Think about it, like all the dot files that you commit to your repo and then you have. So this is one of the disadvantages of a multi repo approach that you need some how to synchronize all these dot files. But with bit you have like your environment component and this way you can share all your configs by using the environment. I wanted to make it easier with PNPM also. So to be able to publish like a package that will contain all your settings. For instance, maybe you use some patches to fix dependencies. If you haven't heard about it, you can use patch files with pnpm. It's also supported in yarn and in NPM you can use a package for that patch packages, I think. So you can if you have some fix for maybe an unmaintained dependency in your node modules and it's very deep in your dependency graph. So you can't just fork it. I mean actually you can fork it and use an override, but with the patch it's even easier because you don't need to publish your fork to the registry. So you generate a patch file and you commit it to your repository and you tell PNPM to use this patch file for the dependency. But if you have many repositories and each repository uses this dependency, you'd have to copy the patch to each repository and make the configuration changes in each repository. With configurational dependencies, you can add this patch file to the configurational dependency and then just use the configurational dependency in each of your repositories and it makes it easier. Also you can for instance have some standard overrides in configurational dependencies. Or you can have the list of package names that you want to hoist to the root of node modules, because sometimes you need that. With pnpm, we also have a feature which other package managers don't have currently, which is called catalogs. It's a way to centralize to synchronize versions of dependencies in a monorepo. So you have these catalogs declared in the PNPM workspace yam file at the root of the monorepo and then inside your projects you can just put catalog column in the version field instead of the version. This way it's easier to control each project in the workspace will use the same versions, which is otherwise possible only using some linting. I think YARN has constraints, it's called which allows to create linting rules for this. And there is a third party package for this I should think called sync. But with catalogs it's a bit nicer because it's centralized in one place and it's nicer to see only change in one file, not in many packages, on files throughout the repository. For the future that we could move these catalogs to this configurational dependencies and maybe companies could have catalogs of recommended versions throughout their repositories, stuff like that. Or even an allow list of dependencies, maybe. I noticed that some companies have limits to what packages they can use for their code base, so it would be easier to ship this.
- [25:48]
C
Yeah, if there's one developer behavior that's well understood in the web ecosystem, it's that people prefer and enjoy configuration files and avoid and hate linting. So it's nice to see that being moved to a first class supported feature. A bitback brought up your company and we should talk a little bit about it because it's fascinating and also very relevant to this area of repo and component management. So you work at bitdev and Bit Dev does use things like pnpm, but why would. Let's say I have a monorepo with a design system and a bunch of apps and some front ends. Why would I want to use something like BitDev or its harmony platform to be able to manage all those components and front ends and so on.
- [26:30]
D
From my point of view, the main advantage is actually this possibility to create your own workspace so your company won't have like to decide whether to use multi package repositories or to use single package repositories or have like multiple monorepositories for different things and then somehow link them together. With Bit, every component is independent. It's cheap actually you can think about it as a single package repository. So each component has its own version control system. But unlike is a git hosted approach, you don't have to like. Bit creates a much better user experience. Even like in the ui the website Bit Cloud. So there is Bit Cloud and Bit Dev. Bit Dev is I think mostly about the open source aspects of bit because you can actually use Bit for free and host it on servers. Because Bit is a an open source project. But then it has Bit Cloud which has additional proprietary code and paid features. But like with Bit, like the code reviews, like when you make changes in multiple components in the pr you will see all the changes to all the components in one place. So that's actually not possible currently with GitHub. Maybe some other companies allow it, I don't know. And it's very well optimized currently for UI components too. So it's like it like has the. How is this project called for UI libraries? I don't remember where you can visually see the components and even like click on them. Storybook. Yeah, so it's kind of like a mixture of storybook and GitHub I think visually. So the problem with Monorepos is that you can't scale it. No matter what you come up with, at some point it will break. You can scale it to that amount of components that you will need in today's world. I know there are like Turbo and NX which improve things a lot for Monorepos, but still at some point it can't handle that amount. So I think GWT is a really unique approach because nobody else tries to solve it this way. Then nobody wants to create a new version control system and a new GitHub. So yeah, just check it out I guess.
- [29:12]
C
Yeah, creating a new git does sound rather daunting. There have been some things built on top of Git parentheses as a service that have sprung up, but it's nice to see the innovation here.
- [29:24]
B
Have you tried building a text to SQL Chatbot? If your AI agents don't understand your data, its definitions, queries and lineage, they're forced to guess. And bad guesses mean risky assumptions. That's where SelectStar comes in. SelectStar automatically builds an always up to date knowledge graph of your data, capturing metadata like lineage, usage and example queries. So whether you're training an AI model or deploying an agent, your AI can answer with facts, not assumptions. Stop the wrong SQL queries before they happen. Learn more@SelectStar.com.
- [30:05]
F
Capital One's tech team isn't just talking about multi agentic AI. They already deployed one. It's called Chat Concierge and it's simplifying car shopping using self reflection and layered reasoning with live API checks. It doesn't just help buyers find a car they love, it's it helps schedule a test drive, get pre approved for financing and estimate trade in value. Advanced, intuitive and deployed. That's how they stack. That's technology.
- [30:27]
C
At Capital One we don't have too much time left for questions, but real quick towards the end I do want to ask you you've Talked about Monorepos vs multi repos or single. You've talked about different forms of package management. Why and how newer innovations in PNPM and Yarn have happened for the next say five years. Are there any more upcoming trends or new areas you're excited about being innovated in.
- [30:53]
D
Well, the CEO of Anthropic said that developers will not be needed in one year.
- [30:57]
C
So I don't know things that are actually going to happen. I should clarify.
- [31:02]
D
To be honest, the problem with a popular project is that you have so much stuff to do that you can't really make a long term. Like you can't think about big stuff. We had many actually like, you know, ban came out right. There is also Origin, which is created by Kat Maran. I hope I pronounce her name correctly, but she was a core contributor to NPM cli. There is also Vault which is created by X NPM maintainers and founders. So sometimes we look at these competitors and think about what could we do differently. So for instance, we spent a lot of time on thinking about rewriting PNPM in Rust or some other language because BUN came out and claimed that it's 10 times faster than PNPM. So to be honest, for a long time I thought that maybe the long term plan would be to rewrite PNPM right in Rust. But then we created a proof of concept and it turned out that the Rust rewrite didn't make it faster. Maybe we just failed, I don't know. But it looked like most of the time is spent on network requests which are not influenced by the text that we use. So I think now my biggest idea for the next few years would be to create some registry integration because we have I think optimized almost everything we could on the CLI part. But if you would have our own registry, for instance, I don't know if we would do it or we would use some existing registry. But if we could like for instance move resolution to the registry side, it could make installation faster or use some different archiving algorithms for the tarables that are faster or make some partial, I don't know. Or maybe we could fetch not the whole packages but just the files that are required from code. So I think it would be interesting to to look at the Registry side. Could be also a way to get some additional funds if you would have like our registry. But the competition is really big currently because I, I know there is JSR from Deno and Vault is coming out and something else, I don't know. Maybe BAM will have its own registry. I don't know. They are very wide in their plans.
- [33:54]
C
Would be unsurprising. We have at Software Engineering Daily an episode from a few months back into 2024, an interview with LUCA from Deno that touches on how yes. Once you push a lot of these concerns into the registry, as you said, you can do a lot of optimizations for them. A big focus is typescript supports and documentation generation. That'd be fascinating. Sultan, I have one last question for you before we wrap up. What is your take on spicy food and where have you had your favorite pieces of hot food?
- [34:24]
D
Thanks. Yeah, I really like spicy food. I'm Hungarian and in Hungary there is a famous soup, goulash, which should be consumed spicy. And there are some sauces, a Hungarian sauce called eru spishto. But then I discovered Asian food, which is even more spicy, and Mexican food. And I really, really love Indian food. I have many friends in India from work, from my previous work. For one week I visited Bengkaluru and there, yeah, it was heaven. A heaven food heaven for me.
- [35:03]
C
Yeah, A lot of folks from North America, from the west, might not know that there are quite a lot of fantastic, very flavorful Eastern European foods such as goulash. And it's lovely to hear that you had such a good time. Was this for a conference that you were in India?
- [35:17]
D
No, it was a work trip. Justanser has an office in India, so I was visiting colleagues.
- [35:23]
C
Well, great. That's all the time. We have one actual final question for you. If people wanted to find out more about you, your projects, anything you're working on, are there any particular locations you'd suggest they go on the Internet?
- [35:34]
D
They can follow me on bluesky, on GitHub on X. Most of my activity is in the GitHub repository in PNPM. We also have a Discord for PNPM, which is quite active.
- [35:47]
C
Well, thank you so much, Sultan, for coming on. It was fantastic talking to you about the history and features of PNPM and package management and workspace and repo and component management. Looking forward to seeing all the great stuff you're doing with the new features, like the dependencies and catalogs for Software Engineering Daily. This is Josh Goldberg and Sultan Kochon. Cheers all. Thanks for listening.
- [36:09]
D
SA.