Show Details

Jon Christensen and Chris Hickman of Kelsus and Rich Staats of Secret Stache conclude their series on the birth of NoSQL and DynamoDB. They compare the NoSQL database, Leviathan, created by Chris’s startup in the late 1990s to today’s DynamoDB. A lot of things haven’t changed, even though technology has evolved. It’s cyclical. There are patterns and problems that continue to dominate.

Some of the highlights of the show include:

Reason for Creation of NoSQL Database: How to scale database with Internet-scale applications to have a virtual pool of infinite storage that can be scaled out
Main Architecture Components of Leviathan:
- API client
- Update distributor (UD)
- Base server (storage node)
- Shepherd (housekeeping management system)
Additional core components included smart IP and storage abstraction layer (SAL)
Leviathan mostly used C code and minimal Java code to support users
Big difference between DynamoDB and Leviathan is request router and partition metadata system living on the server vs. living on the edge
Leviathan was a closed system with an instance for every network or data center; not designed to run as a software as a service, like DynamoDB
Leviathan was strongly consistent, unlike DynamoDB’s eventually consistent model
Definition and Different Types of Transactions
Shepherd was used to identify and address consistency, synchronous, and timing issues
Rather than using a file system, Leviathan used relational databases

Links and Resources

Quotes:

“We had the same kind of problems that DynamoDB had - how do you scale your database dealing with Internet-scale applications and have this virtual pool of infinite storage that can be scaled out.” Chris Hickman

“This system and this technology went through many iterations.” Chris Hickman

“You can’t have a 100% consistent state across everything. It’s just impossible. How do you do the right thing?” Chris Hickman

“The big difference between DynamoDB and Leviathan...is the request router and partition metadata system living on the server vs. living out at the edge.” Jon Christen