Podcast Summary: Building PostgreSQL for the Future with Heikki Linnakangas

Episode: Building PostgreSQL for the Future with Heikki Linnakangas
Host: Kevin Ball
Release Date: May 20, 2025
Podcast: Software Engineering Daily

1. Introduction

The episode kicks off with an overview of PostgreSQL's reputation as a robust, extensible, and SQL-compliant open-source database. Heikki Linnakangas, a prominent PostgreSQL developer and co-founder of Neon, joins host Kevin Ball to delve deep into PostgreSQL's enduring popularity, its extensibility, the role of extensions like PGvector, and the innovative serverless platform Neon.

2. PostgreSQL's Popularity and Longevity

Heikki Linakangas reflects on PostgreSQL's journey, noting its transformation from a lesser-known database in the early 2000s to the default choice for many developers today.

"It's become the default. Which is funny, it feels strange to me because it used to not be that way, but nowadays people just take it for granted that people are using postgres."
[02:17]

He attributes PostgreSQL's longevity to its stability, comprehensive feature set, and a disciplined release management process that ensures predictable and regular updates. The extensive ecosystem, rich tooling, and the ease of finding solutions through extensive documentation and community support further consolidate its position.

"It has a big ecosystem of all kinds of tools... a lot of people are like, there's a big ecosystem and if you have a problem with Postgres, you can Google that easily."
[04:25]

3. Extensibility in PostgreSQL: Core vs. Extensions

A standout feature of PostgreSQL is its unparalleled extensibility. Heikki traces this back to Postgres's origins as a university project focused on flexibility.

"Postgres has a very flexible type system. You can create your own data type with its own functions, and not just functions, but also operators."
[04:55]

Heikki emphasizes that PostgreSQL's ability to handle custom data types, operators, and indexing systems enables a wide range of applications, from geographic information systems with PostGIS to AI-driven applications using PGvector.

"Postgres goes a lot deeper in that. Like even all of the built in data types are created using the same primitives... you can build your own indexing system for that."
[04:55]

The distinction between what resides in the core database and what exists as an extension is crucial. Heikki advocates for keeping the core lean, allowing extensions to innovate independently without burdening the main PostgreSQL codebase.

"Extensions have a lot of advantages. Like you can have your own release schedule, you get to decide your own releases, you get to decide your own other external dependencies."
[08:13]

4. PGvector and AI Applications

The conversation shifts to PGvector, a popular PostgreSQL extension tailored for handling vector data essential in AI applications.

"PGvector uses the same algorithm as many other vector databases, HCNs, W and IVFlat."
[11:14]

Heikki discusses the challenges and innovations brought by PGvector, such as handling large-dimensional vectors, the CPU-intensive nature of building vector indexes, and the absence of efficient disk-based algorithms, which necessitates in-memory processing for optimal performance.

"What I have done, when I started to look at PGvector, I started to look at it from the point of view of I know indexes... it's very different from traditional data types."
[13:15]

A notable point is the approximate nature of vector searches, which contrasts with PostgreSQL's foundational goal of exactness in query results.

"Whenever you do a search on a vector index, it's always like the thing it does... they are always approximate."
[13:28]

This introduces complexities when integrating approximate results within traditional SQL queries, posing challenges in maintaining SQL's guarantee of exact results.

5. Neon: Building a Serverless PostgreSQL Platform

Heikki introduces Neon, a serverless platform for PostgreSQL designed to separate compute and storage layers, inspired by Amazon Aurora's architecture but fully open-source.

"The core is the separation of compute and storage. So the idea is that there is a separate storage system which keeps all of the history."
[17:07]

Key features of Neon include:

• Separation of Compute and Storage: Allows independent scaling and efficient resource utilization.
• Point-in-Time Recovery: Enabled by maintaining a comprehensive history of the write-ahead log (WAL), replacing traditional backup methods.
• Serverless Architecture: Achieved by decoupling persistent storage from compute, enabling rapid provisioning and scalability.

Heikki elaborates on how Neon uses a unique storage layer that processes PostgreSQL's WAL to reconstruct any version of a data page, facilitating features like read replicas and point-in-time queries without duplicating data.

"We can just tell it to please show me all the data at this older point in time... the storage layer can do that."
[22:22]

6. Future of PostgreSQL and Neon

Heikki shares insights into the future roadmap for PostgreSQL, highlighting upcoming features like asynchronous I/O and multi-threading.

"There's no central roadmap for postgres. So it always depends on what other stuff that people submit patches for."
[09:55]

He envisions PostgreSQL evolving to better accommodate serverless environments, addressing challenges like connection management and memory allocation. The transition to multi-threading is seen as pivotal for enhancing scalability and flexibility.

"I would love to see much more flexibility in Postgres so that it's easier to run in this kind of serverless environment."
[35:18]

Regarding Neon, Heikki outlines ongoing efforts to optimize cold start times, currently averaging around one second, and plans to further reduce this latency by maintaining a pool of pre-warmed virtual machines.

"We have plans to bring it down further and further. But it's not really a problem anymore."
[20:00]

7. Open Source Community and Contributions

The strength of PostgreSQL lies in its vibrant open-source community. Heikki encourages developers to contribute, emphasizing that contributions often stem from solving personal challenges or innovating new features.

"All of people historically have gotten active with Postgres because they have an itch to scratch."
[40:15]

For those interested in contributing, he suggests starting with writing extensions, which provides a practical avenue to engage with PostgreSQL's extensible architecture.

"Writing your own extension is a good way to get started. We talk about all the extensible type systems and stuff."
[40:42]

Neon's storage layer is also open-source, allowing the community to explore, modify, and deploy it independently, fostering broader innovation.

"The Neon storage layer is all open source... If somebody wanted to take the Neon storage system and run it somewhere else, they could just go and do that."
[31:38]

8. Closing Remarks

In wrapping up, Heikki reiterates his optimism for PostgreSQL's future, likening its enduring presence to Linux's dominance in the open-source ecosystem. He underscores the collaborative spirit of PostgreSQL's community, advocating for ongoing contributions to drive the platform forward.

"Postgres is still alive and kicking. Like there's not... it has some staying power in that, like just being the lowest common denominator between all of the forks and all of the different approaches."
[38:02]

Host Kevin Ball expresses enthusiasm for Neon's capabilities, particularly its streamlined approach to managing data persistence and point-in-time queries, highlighting its potential to revolutionize how developers interact with PostgreSQL.

Key Takeaways:

• PostgreSQL's Strengths: Stability, extensibility, and a strong community underpin its lasting popularity.
• Extensibility: PostgreSQL's core vs. extensions model fosters innovation while keeping the main codebase lean.
• Neon: A pioneering serverless PostgreSQL platform that decouples compute and storage, enabling scalable and efficient database management.
• Future Directions: Embracing multi-threading and enhancing serverless capabilities are pivotal for PostgreSQL's evolution.
• Community Engagement: Active contributions and the open-source ethos are crucial for PostgreSQL's continued success.

Notable Quotes:

• "It's become the default. Which is funny, it feels strange to me because it used to not be that way, but nowadays people just take it for granted that people are using postgres." — Heikki Linakangas [02:17]

• "Postgres has a very flexible type system. You can create your own data type with its own functions, and not just functions, but also operators." — Heikki Linakangas [04:55]

• "Whenever you do a search on a vector index, it's always like the thing it does... they are always approximate." — Heikki Linakangas [13:28]

• "I would love to see much more flexibility in Postgres so that it's easier to run in this kind of serverless environment." — Heikki Linakangas [35:18]

• "Postgres is still alive and kicking... it has some staying power in that, like just being the lowest common denominator between all of the forks and all of the different approaches." — Heikki Linakangas [38:02]

This episode offers a comprehensive exploration of PostgreSQL's enduring legacy, its flexible architecture, and the innovative strides being made through platforms like Neon. For developers and database enthusiasts, it underscores the importance of community-driven development and the endless possibilities that arise from PostgreSQL's extensible framework.