AWS Podcast Episode #704: Scaling Your Relational Database on AWS – Are Relational Databases Cool Again?

Release Date: January 20, 2025

In Episode #704 of the AWS Podcast, titled "Scaling Your Relational Database on AWS – Are Relational Databases Cool Again?", hosts Lish and Josh Hart, a Principal Solutions Architect at AWS based in the UK, delve into the evolving landscape of relational databases. They explore the challenges of scaling relational databases and introduce innovative AWS solutions designed to address these issues. This comprehensive discussion is invaluable for developers and IT professionals seeking to optimize their database infrastructure on AWS.

Introduction to Relational Databases and Scaling Challenges

The episode begins with Lish highlighting the enduring relevance of relational databases in modern applications, despite the rise of NoSQL databases. She shares her experience with DB2 mainframe databases, emphasizing their reliability and fit for numerous solutions. However, she acknowledges the inherent scalability issues that often prompt organizations to consider alternatives like NoSQL.

Lish (00:37):

"Relational databases are at the core of many applications and have been for many, many decades. They work and fit into a lot of solutions."

Josh concurs, pointing out that as applications grow, relational databases can become bottlenecks. He explains that simply adding more compute resources ("throwing more tin") is a short-term fix that eventually hits physical and practical limits.

Josh Hart (02:02):

"Very often, you see, the relational database can become the bottleneck and it can become the sticking point as you start to want to scale your application."

Vertical vs. Horizontal Scaling

The conversation shifts to the two primary methods of scaling databases:

1. Vertical Scaling: Increasing the resources of a single database instance.
2. Horizontal Scaling (Sharding): Distributing the database across multiple nodes.

Lish (04:02):

"If we can't keep going big, let's say, in terms of a single node, what's stopping us from having lots of nodes like you mentioned auto scaling before, why don't we just autoscale the database?"

Josh elaborates on the complexities of each approach. Vertical scaling often requires downtime or failover during resource upgrades, making it inflexible for dynamic workloads. Horizontal scaling introduces application-level complexity through sharding, which necessitates managing multiple database instances and routing queries appropriately.

Josh Hart (05:12):

"Sharding is where we have to make the application aware that our database is sharded... that's complexity that we're adding to the application."

AWS Solutions for Scaling Relational Databases

To address these challenges, AWS has developed solutions that simplify both vertical and horizontal scaling:

1. Aurora Serverless v2

Aurora Serverless v2 allows automatic scaling of database resources in response to fluctuating workloads without requiring restarts or causing downtime. This solution is particularly beneficial for environments with variable usage patterns, such as development and pre-production.

Josh Hart (07:13):

"Aurora serverless V2... will automatically scale up and down in response to changing traffic patterns... seamlessly without having to restart the database process."

Key features include:

• Rapid Scaling: Responds to load changes within seconds.
• Cost Efficiency: Scales down during low usage periods, reducing unnecessary costs.
• High Availability: Maintains uninterrupted service during scaling operations.

Lish (09:44):

"It will scale down more conservatively... you can begin to procure additional capacity... typically seconds."

2. Aurora Limitless Database

Aurora Limitless Database, currently in preview, enhances horizontal scaling by abstracting the complexities of sharding. It introduces "shard groups," which automatically partition data across an expandable set of nodes based on predefined shard keys.

Josh Hart (12:08):

"Aurora Limitless Database... automatically manage the routing of the query to the correct storage node. So your application doesn't need to be sharding aware."

Benefits include:

• Simplified Application Logic: Applications connect to a single Aurora cluster endpoint without needing to manage shard-specific connections.
• Automatic Partitioning: Data is seamlessly distributed as the database scales horizontally.
• Reduced Operational Overhead: Eliminates the need for manual sharding management and routing components.

Lish (13:41):

"That means we're taking away the undifferentiated heavy lifting of sharding your database. It's done and done robustly."

Connection Management Challenges and Solutions

Beyond scaling, managing database connections is a critical aspect of database performance. Traditional relational databases can struggle with connection storms, which occur when applications rapidly open and close connections, leading to resource exhaustion.

Josh Hart (15:24):

"Each connection that we have to the database consumes some additional resource, some CPU, some memory... connection churn can have a real material overhead."

AWS offers two primary solutions to manage connections effectively:

1. RDS Proxy

RDS Proxy acts as an intermediary between applications and databases, controlling the number of active connections and enhancing database resilience.

Josh Hart (17:04):

"AWS solution there is called RDS Proxy... you have to manage the configuration of."

While effective, RDS Proxy requires provisioning and managing an additional layer, which can introduce complexity.

2. RDS Data API

A more recent innovation, the RDS Data API, reimagines connection management by eliminating the need for persistent database connections. Instead, it leverages SDK calls to interact with the database, handling connection pooling internally.

Josh Hart (17:49):

"RDS Data API will manage the connection pooling for you as part of the service, and all you need to do is just send your SQL statement as an API call."

Advantages include:

• No Persistent Connections: Reduces resource consumption by avoiding open TCP connections.
• Simplified Development: Developers interact with the database through standard API calls, removing the need for complex driver configurations.
• Enhanced Security: Eliminates the need for VPNs or open ports, allowing secure access from any environment with proper IAM credentials.

Lish (19:11):

"I'm sending my SQL... but I'm not configuring ODBC drivers or JDBC drivers and all that sort of plumbing again, kind of abstracted away."

Business Implications and Operational Efficiency

Lish and Josh emphasize the business benefits of these AWS solutions, highlighting reduced operational overhead and cost savings. By automating scaling and connection management, organizations can redirect their resources towards innovation rather than database maintenance.

Josh Hart (20:57):

"This is going to reduce the amount of operational overhead... you can focus on the innovation and the thing that really kind of differentiates you."

Lish (21:57):

"Look at your dev test environments... something like serverless, Aurora Serverless 2, you can have the size you need within reason to get the job done."

These advancements align with a common organizational goal: minimizing the time and resources spent on "keeping the lights on" in IT operations, thereby maximizing investment in innovative projects.

Conclusion

The episode wraps up with Lish and Josh reinforcing the importance of leveraging AWS's advanced database solutions to overcome traditional relational database challenges. By adopting Aurora Serverless v2, Aurora Limitless Database, and the RDS Data API, organizations can achieve scalable, efficient, and cost-effective database architectures that support both current and future application demands.

Lish (23:32):

"Thanks everyone for listening... Until next time, keep on building."

Key Takeaways:

• Relational Databases Remain Vital: Despite the popularity of NoSQL, relational databases continue to underpin many critical applications.
• Scaling Challenges: Vertical scaling has limitations and can lead to inefficiencies, while horizontal scaling introduces complexity through sharding.
• AWS Innovations:
  • Aurora Serverless v2 offers seamless vertical scaling without downtime.
  • Aurora Limitless Database simplifies horizontal scaling by automating sharding and query routing.
  • RDS Data API redefines connection management, enhancing scalability and security.
• Business Benefits: These solutions reduce operational overhead, optimize costs, and allow organizations to focus on innovation rather than database maintenance.

For developers and IT professionals looking to optimize their relational databases on AWS, Episode #704 provides essential insights and actionable solutions to modernize and scale database infrastructures effectively.