Podcast
The Backend Engineering Show with Hussein Nasser
Hosted by Hussein Nasser · EN
Welcome to the Backend Engineering Show podcast with your host Hussein Nasser. If you like software engineering you’ve come to the right place. I discuss all sorts of software engineering technologies and news with specific focus on the backend. All opinions are my own.
Most of my content in the podcast is an audio version of videos I post on my youtube channel here http://www.youtube.com/c/HusseinNasser-software-engineering
Buy me a coffee
https://www.buymeacoffee.com/hnasr
🧑🏫 Courses I Teach
https://husseinnasser.com/courses
11episodes
Episodes
Newest firstAll episodes
Postgres is half as fast in Linux 7.0
3w ago00:34:08Tap to summarizeAn aws engineer discovered a 50% regression in postgres throughput while testing the new Linux 7.0 kernel. The cause turns out to be massive TLB and page faults exacerbated by Postgres process-based design. In this backend engineering show episode I dive deep into how this was discovered, the root cause and the possible fixes and workarounds. Intermediate and Advanced Backend Engineering Course Bundlehttps://courses.husseinnasser.com/bundleMy Book, Root Cause: Stories and Lessons from Two Decades of Backend Engineering Bugs https://amzn.to/4cKfZhe 0:00 Intro2:30 The Discovery6:30 Spinlocks9:25 Preemption 13:00 Root Cause17:00 How Postgres Processes exacerbated the problem 22:30 Is the fix easy?25:50 Summary
Transcribe →Don't let AI rob you
4w ago00:31:53Tap to summarizeA discussion about why many engineers still love the struggle, the mistakes, and the process of figuring things out themselves. This is how we grow and get better and stronger. Letting AI do everything (even though it can’t) robs us this feeling..
Transcribe →New Book: Root cause
Apr 1500:09:48Tap to summarizeI wrote a new book that has been in the works for years. It is called Root Cause, and it is for those who enjoy the art of backend engineering.Early in my career, 20 years ago, I built backend and database applications without fully grasping their inner mechanics. Performance issues, race conditions, bugs, and even data corruption often left me lost.Since that day, I resolved to truly understand how systems work. From networking protocols and intermediary proxies to backend services and various database engines. I made it a habit to follow every request on its journey through the dark alleys of the network, down to the bowels of the database engine, meanwhile interacting with various kernel data structures in the process at every hop, and back.I became obsessed with understanding what happens behind the scenes in software. Not just what breaks, and how but also why and what was the source of the bleed. Root Cause is a collection of the most interesting bugs I encountered, ranging from performance bottlenecks and non-deterministic crashes to subtle data inconsistencies and incorrect results.This book is for anyone curious about how production backend systems really behave under pressure, and how to debug them when they don’t. Even when you don’t have access to the source code.Root cause consists of 15 chapters, each is a story about a backend bug, with investigation, diagrams, a section of a fundamental concept until the root cause is revealed.Grab your copy here paperback or kindle ebookpaperbackhttps://amzn.to/4cKfZheebookhttps://amzn.to/4cfQjJj
Transcribe →5 Backend Design Patterns for Managing Threads and Sockets
Jan 1900:46:09Tap to summarizeIn this video I introduce 5 different design patterns for building backend applications. Each mode explains how a socket listener is established, a connections are established and how threads and connections are managed to read, write and process requests.
Transcribe →Page Tables
Dec 1500:46:39Tap to summarizePage tables provide the mapping between virtual memory and physical memory for each process. This means it needs to be as efficient and as fast as possible. I explore the inner workings of page tables in this episode.0:00 Intro2:00 Virtual Memory ⁃ ⁃ 8:00 MMU10:00 Page Tables ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ 11:30 Single Table Byte Addressability ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ 16:00 Single Table Page addressability ⁃ ⁃ ⁃ ⁃ ⁃ 19:00 Multi-level Paging (Radix tree) ⁃ ⁃ 31:00 Huge Tables ⁃ ⁃ 33:00 TLB ⁃ ⁃ Summary
Transcribe →CPU and Kernel Page Faults
Nov 2400:48:37Tap to summarizePage faults occurs when the process tries to access a memory that isn’t backed by a physical page kernel raises a fault which loads a page. It happens on first access, stack expansion, COW, swap and much more. However it comes with a cost. In this episode of the backend engineering show I dissect the need and the cost page faults in the kernel. 0:00 Intro 4:00 Virtual memoryAbstraction of physical memoryMemory sharingAllow more processes to run , unused go to diskNuma, kernel can place memory near the cpu12:00 VMA areasText/code Data BSSHeapStack19:50 Kernel mode25:30 What is a Page fault?30:30 First access page fault33:00 Stack Expansion page fault34:30 CoW page fault38:00 Swap page fault39:39 File backed page fault40:29 Permission page fault 45:30 Summary
Transcribe →Amazon US-EAST-1 Outage in Details
Oct 3100:24:26Tap to summarizeOn October 19 2025 AWS experienced an outage that lasted over a day, 10 days later we finally got the root cause analysis and we know exactly what caused the DNS to fail0:00 Summary 5:30 How did Dynamo lost its DNS?13:41 EC2 Errors 16:16 Network Load Balancer ErrorsRCA here https://aws.amazon.com/message/101925/
Transcribe →Graceful shutdown in HTTP
Oct 1700:25:49Tap to summarizeThere are cases where the backend may need to close the connection to prevent unexpected situations, prevent bad actors or simply just free up resources. Closing a connection gracefully allows clients and backends to clean up and finish any pending requests. In this episode of the backend engineering show I discuss graceful connections in both HTTP/1.1 via the connection header and HTTP/2 via the GOAWAY frame. 0:00 Intro4:58 Why shutdown connection? 6:46 HTTP/1.1 Graceful shutdown12:26 Cost of HTTP/2 17:40 HTTP/2 GoAWAY frame23:40 SummaryLinkshttps://www.youtube.com/watch?v=fVKPrDrEwTI&t=1s https://chromium.googlesource.com/chromium/src/net/%2B/master/socket/client_socket_pool_manager.cc#76https://issues.chromium.org/issues/40555364https://issues.chromium.org/issues/40501721
Transcribe →Postgres 18 gets Async IO
Oct 300:41:12Tap to summarizePostgres 18 has been released with many exciting features such as UUIDv7, Over explain module, composite index skip scans, and the most anticipated asynchronous IO with worker and io_uring mode which I uncover in this show. Hope you enjoy it0:00 Intro1:30 Synchronous vs Asynchronous calls3:00 Synchronous IO6:30 Asynchronous IO10:00 Postgres 17 synchronous io 17:20 The challenge of Async IO in Postgres 1820:00 io_method worker23:00 io_method io_uring29:30 io_method sync 31:08 Async IO isn’t done! 31:30 Support for backend writers32:36 Improve worker io_method33:00 direct io support 37:00 Summary
Transcribe →Kernel level TLS
Jun 13, 202500:22:55Tap to summarizeFundamentals of Operating Systems Course https://oscourse.winktls is brilliant.TLS encryption/decryption often happens in userland. While TCP lives in the kernel. With ktls, userland can hand the keys to the kernel and the kernel does crypto. When calling write, the kernel encrypts the packet and send it to the NIC.When calling read, the kernel decrypts the packet and handed it to the userspace. This mode still taxes the host’s CPU of course, so there is another mode where the kernel offloads the crypto to the NIC device! Host CPU becomes free. Incoming packets to the NIC are decrypted in device before they are DMAed to the kernel. outgoing packets are encrypted before they leave the NIC to the network.ktls still need handshake to happen in userspace. There is also enabling zerocopy in some cases (now that kernel has context) Deserves a video. So much good stuff.0:00 Intro2:00 Userspace SSL Libraries 3:00 ktls 6:00 Kernel Encrypts/Decrypts (TLS_SW)8:20 NIC offload mode (TLS_HW)10:15 NIC does it all (TLS_HW_RECORD)12:00 Write TX Example13:50 Read RX Example17:00 Zero copy (sendfile)https://docs.kernel.org/networking/tls-offload.html
Transcribe →