C (2:59)

Jillian, you had my friends John and Tarik on a couple episodes ago talking about this very topic, right?

B (3:06)

We did, yes. And I'm so glad that you made a plug for that episode. So folks, if you missed it, definitely listen to that one as well if you really want to just master Multi AZ before getting into the multi region topic. So I love that. So, okay, so they've looked at Multi AZ with and they feel like they've got that down within a single region. And I know a lot of customers, they start to really think about it maybe from a business requirement, which totally makes sense. Maybe they've got, they're expanding into a different part of the world or maybe there's compliance requirements that require for them to think about multi region. So now that they've decided, okay, check the boxes, it makes sense to actually go multi region. What are some additional prerequisites that customers should think about in order to be able to approach a multi region architecture?

A (4:06)

I think one of the really important things is to understand the logical and physical separations between AWS regions.

B (4:14)

Right.

A (4:15)

It's going to require architecting your application in a specific way to not lose the benefits of that, of that logical and physical isolation. You don't want to build a multi region app that still experiences downtime if there's an impairment in either one of the regions that you're running in. So maintaining that strict isolation can be challenging. It really requires thinking of your application architecture in a different way. You know, multi region give you that, that kind of bounded recovery context. So you don't want to architect your way out of the benefits of, of that separation that we've created between our regions.

B (4:57)

Yeah. Can you elaborate on that, of how customers can think about that, those tight isolation boundaries? Because I could totally see that being a common mistake, especially maybe if you're someone that is new to the cloud. Yeah.

A (5:12)

You know, I think, you know, if you think about failing an application between regions, having that separation between your application stacks, they really need to be able to stand alone, you know, to survive an impairment in, in either space, any of those dependencies, you need to fail. When you fail from one region to another, you need to make sure that everything is failing over altogether. Right. That, that requires just a lot of operational muscle, a lot of, a lot of planning and coordination. It'll be coordination between your engineering teams, the business. There's a, there's, it's more than just a technical challenge when you're designing these types of systems. We often will say that, you know, the culture of resilience is an important part of being able to succeed with these larger, you know, more complex multi Region infrastructures, you know, kind of having that built into your, your engineering practices, having that OPER muscle to regularly practice failover between regions or, or you know, simulating outages of various components, being able to make sure that you understand how your system's going to behave in those, in those scenarios. And it also requires thinking a little bit differently about how you're managing your data. Right. If you know, you're looking at things like consistency versus availability. Right. The standard cap theorem, trade offs, partitioning is part of the, the multi region infrast. So now you're left with the decision between consistency and availability. So really understanding where your priorities lie and engineering your architecture to support the requirements of your application.

B (6:59)

Yeah, definitely. I think you bring up a lot of interesting points that I think then causes for people to realize, okay, there's actually a lot that needs to go into planning and this out. So for customers where this is their first time going from 1 to 2 AWS regions, walk me through like some suggestions how you would think about that customer should go about that process.

A (7:29)

Yeah, I think first, you know, a full understanding of your application architecture is just a table stakes requirement. Right. And then you know, building a runbook, you know, don't just rely on everybody getting things right in the moment. You've really got to have everything kind of documented and understood. You've got to know when your components, you know, come up in a different region. You've got to understand what are all the dependencies because you can't some, you know, if you're going with like an active passive model and you're doing an actual failover, then coming up in another region requires that all of your components have been, you know, been brought online before you are able to fully start running your application in that secondary space. Right. And with data replication you've got to decide if you're going to use an asynchronous or a synchronous approach. There are differences in the time that it takes, the data latency because there's a need to copy that data between regions. So making sure that you aren't out of sync. If you're using synchronous replication across regions, then you've got to avoid issues with your data being out of sync when you've started your application, that new region, but with the latent latency characteristics, you know, your right commits need to, need to be accounted for in your, in your cross region design.

B (8:50)

Yeah, it sounds like there's really definitely a lot that I think customers need to think about and so I can imagine that even like once you've actually done the design and have it actually in production, there's still a lot that needs to be managed in terms of just operating it. So Scott, I'm curious, from your perspective, what are some of the challenges that you've seen customers face when they're operating multi region applications?

C (9:18)

I think maybe take a step back. The customers who are thinking about moving to multi region or starting from multi region at the beginning, the first thing that we tend to talk about is calculating how, how much downtime would actually cost them. So if you were in a single region and your application was to be unavailable, how much does this cost you per minute of outage? And then when you consider the move to multi region or the build in multi region, how much is it going to cost to have this redundancy that Jeff's talking about in multiple regions? And what we find is that usually the redundant cost is kind of outweighed by the cost of being unavailable entirely. So it just helps frame the question a little bit for me. Can you afford to not be available? What challenges are there once you are in multi region Coordination? Resonates with me a lot. You see multiple engineering teams engaged on a single operational event. It's hard or it can be difficult for them to communicate state between the teams to get the right approvals online when they need to be cross account. Operating cross account can be challenging. It just increases the number of eyes that you need on the problem and makes it difficult to understand the status of your recovery.

B (10:51)

Yeah, and it sounds like there's just a lot that really needs to go into to actually be able to operate like with these multi region applications. So I'm curious, Jeff, what are some strategies that you've seen customers do up, up until this point?

A (11:09)

So I think, you know, having a solid communication plan is important. I like to talk about some of the things that we do at, at Amazon because they were very unique to me. And I've been here for 11 years, so I've seen this all in motion here at Amazon for years. And I still remember the very first operations call that I jumped on, you know, way back in the beginning. Everyone at Amazon is invited to listen in on these operations calls. And I thought that was very strange. I was in professional services at the time. It made no sense for me to be participating in a call about, you know, things that were going on in the, the operational business of, of aws. But then I really started to understand the reason why everyone's invited. Right. It, it makes sure. That as we're building things, we understand not only how is, you know, how our own stacks are built, but we understand the, the, the issues that people have, have run into that came before us. The other part of that call that was really fascinating to me was we talked about the operational wins. This wasn't just a call to talk about issues or concerns, but what went right, you know, what were some good things that happened and there are there practices or policies that could come out of that that would be valuable to other engineering teams. So that really introduced to me the idea of resilience as a core part of a company's culture. It was a, it was a blameless call. And they did talk about things that hadn't gone quite right. No one ever asked whose fault it was. They just wanted to understand how do we prevent these things from happening again. There was another very interesting part of that where that we call the wheel spin just never seen anything like it. Where just at random, two different groups were asked to explain their, their current operational metrics. Now they weren't having any issues with their, their services, but the idea is by always needing to be ready to look at your operations dashboard and explain what's going on, you kind of build that muscle so that you're never, you're never letting things just kind of run on their own. Right. People were always aware of the latest details for their service. If they were called on during the wheel spin, they had to talk about, you know, any, any deviations in performance, whether that was positive or negative, and did they understand why those things had happened. And that just really reinforced how deeply ingrained in the culture that operational element was. Right. And I think that's a, a great way to make sure that you are constantly working on those communications channels, constantly understanding the, you know, operational status of your service, whether it's good, understanding where you have visibility or where you don't have visibility, where you have gaps in your observability and those sorts of things come out through those sort of regular practices. It was just, it's still, I joined those calls anytime I can. You know, we have, it's about a two hour call every week and you know, I probably make one a month at least. And it's just really informative and really interesting to see how that communication flows and how other teams are doing their observability and monitoring.

C (14:28)

I've been on the receiving end of that wheel spin, Jeff, a number of times and I'll tell you, it is, it is an interesting experience and observability definitely a requirement in order to answer the sort of questions that you would get on a call like that. And that actually makes me think of one more thing that customers have sometimes have challenges with during their multi region journeys, which is providing evidence that they're in compliance with regulatory restrictions. You know, whether it's data sovereignty or similar items, it's the fog of war during these operational calls can make it difficult to piece together the event after the fact, especially if you're lacking that observability. This is a challenge. We hear from folks all the time.

B (15:19)

It's such an interesting process that Amazon does and I'm sure a lot of customers who are listening right now are like, wow, that sounds really cool. How could I do something similar? Like, I mean I'm, I'm not Amazon, but I definitely want to make sure that I'm, my application's like resilient. So do you have any suggestions of how to apply that type of like meeting structure to like a smaller IT team?

A (15:46)

Yeah, I'll start by saying it's not going to happen overnight. Right. This is, it is a significant difference from the way many companies run their, their ops calls and if you've already got a good strong operations culture, you're, you're a step ahead. But it really has to be driven from the top down. Right. This is, this is an executive level decision. It has to become just a, you know, executive sponsored component that, that has intent behind it. It's not going to happen by accident. Right. It has to be driven from the top and it has to be a priority of all the teams involved, from the engineering teams to the business teams to your finance. Some of these things. There's cost in taking two hours out of every engineer's day to jump on these calls. But there's value in jumping on those calls and I think it's critical to understand that that value is there. It is worth the cost to invest that time in building an operational muscle.

B (16:43)

Scott, I'm curious, from your perspective, having been on the receiving end of those calls, do you have any advice for smaller IT teams of how they can implement like a similar process?

C (16:55)

I think Jeff's, Jeff's summary is pretty good. It's an investment, but the way I think about it is you're either spending that time preparing for an event or you're taking that additional time during an event when your customers aren't able to access your application. So I do think it's a worthwhile investment. Top down sounds right. I also feel like a couple Motivated engineers or managers can start this work on their individual teams, start building up best practices there and, you know, results tend to speak for themselves. This sort of thing catches on with other teams. Build culture from the bottom up.

B (17:37)

Yeah, I love that. It sounds like even though it's a big time, I say that with air quotes. Even though people can't see me time investment, it actually ends up simplifying your operations long term because you're doing so much planning ahead of time. So I like that from like the like people culture, operations part. I'm curious, Scott, like, is there anything else that AWS is doing that can actually help customers with their entire infrastructure to be able to manage this, like regional switch?

C (18:16)

I would say, well, just a couple mechanisms that I really like and then I'll talk about a new service that we are, that we've launched, that we're pretty excited about. But the mechanisms that I think are pretty good. One of them is called orr. And what this is is kind of a pre mortem for your service where you can look at an application and say, here is 50 things that we've seen gone wrong in the past. How have you addressed those 50 things? Like, are you prepared to actually take on traffic from a standby region if there was to be an issue? Can you provide that evidence for preparation? So think of it as a pre mortem. And then the other mechanism that I'm a big fan of is a kind of a postmortem called coefficient, which is correction of error. And what this is is it's a extremely deep dive into an operational event where we say, why did this happen? Okay, why did that happen? We keep asking ourselves why until we get to the root cause of what the problem was. It really shines a light on events or root causes of events after they occur and helps you prevent those in the future. So I'd say that pre mortem and postmortem in combination are pretty easy to roll out and pretty effective.

B (19:39)

So you can think of it as.

C (19:40)

A list of best practices for building and operating applications that are built up over time and informed by actual issues that happened in production. And a team would take this list and while they're building their application, they would check it, say, oh, you know, we have a S3 bucket over here. Do we have the right level of encryption? On that bucket is our. Have we made sure that we're scaling appropriately for compute in a redundant region that we operate in? Things like that that, you know, most teams would just sort of do over time. But it's a checklist to make sure that no one forgot something. So I think of it as a pre mortem making sure that what you're building will stand up to issues from the past. The post mortem mechanism we call coefficient, which is after a live event has occurred, we really spend a lot of time thinking about how this happened. Not just what the first event or what the first issue was, but what caused that first issue and then what caused that issue. And we drill down until we find the root cause of the problem. Then we talk about all the ways that we can remediate this and make sure it doesn't happen again in the future. Once we find enough of these frequent issues, those become pre mortem questions. So it's this little flywheel of success for operations that it's really just a couple meetings and some deep dives on events.

B (21:12)

Wow. Yeah, it's super interesting. I think a lot of people I hope at least are going to expire to want to implement a form of that, just like pre mortem as part of their own operations process. But Scott, you were selling us earlier that there's this new service that's come out.

C (21:33)

There is. I'm very excited about this one, Gillian. So, like I mentioned, I work with a team called Amazon Application Recovery Controller and we have a number of services that help with multi region and in region recovery. But these are. You can think of these previous solutions as kind of triggers for recovery. You flip an on off switch to help move traffic away from an impacted area. The problem is with a multi region application, like Jeff was saying, it's very complicated. There can be a lot of dependencies, there's a lot of people on the call who are trying to recover the application. Coordination can be hard. So what we've built is a new service that orchestrates recovery for multi region applications and takes some of the concepts that we've already built, takes concepts from other teams within Amazon, within AWS and centralizes all that. So teams can just go to one place and work through their recovery journal or work through their recovery journey.

B (22:40)

Yeah, can we. Yeah, I was just gonna say like if you can elaborate on that more about like being able to, to coordinate recovery. Especially if like, like how do you like the combination of having that orchestrated? So it's not solely relying on just like human intervention versus maybe you might want some human intervention at some point. So maybe like how does Application Recovery Controller, Region switch like help in both scenarios?

C (23:09)

Yeah, let me, I'll talk your ear off about this for a minute. I'll Just put it all out on the table. So the service is called Region Switch. That's what we launched. It supports active passive and active active applications. So either one of those slots right in. We built it to address some of the common issues that Jeff had mentioned. Like, customers traditionally had written custom scripts that span multiple accounts, and those are hard to coordinate during a live event. It's hard to orchestrate recovery of multiple resource types in a single spot. Like, you might need to fail over your compute resources and your databases, but those are. They tend to be two separate swim lanes. That's extra coordination. It's difficult to actually test that your recovery mechanisms are working correctly. You tend to find that they're broken during a live event, which is no good. So we've built a service that we think will help customers address this. So how you can think about it is we have a concept called recovery plans. And a recovery plan, you could tie one to one with an application, and the recovery plan is the record of all the steps that need to be taken to recover this application during a regional impairment. So within that plan, we have a concept called workflows. And you can think of workflows as a flowchart. In fact, you can create a workflow using a drag and drop GUI from the console to create that flowchart, or you can put it in infrastructure as code. Like, it's pretty flexible there. Once you define that workflow, what you're doing is you're setting execution blocks, is what we call them, and you can think of these as building blocks for recovery. So at launch, we had nine, I believe, nine execution blocks. And these help you scale your application, trigger traffic, shifting between regions for your application. We have manual approval steps. So this gets back to your question about when a person would be involved. So you might say you'd recover your database layer and then wait for a senior engineer to make sure everything works and manually approve it. We also support custom recovery actions through Lambda. So chances are folks already have some set of recovery steps that they take today. We allow you to bundle that up in a lambda and execute that way. One of the more powerful things that we do as well is we have Region Switch Plan Orchestration, which you can think of as a plan of plans. So as you're recovering, if you have dependent applications, those become little forks in the road of the top level application recovery plan. Yeah, I have a lot more to say about this, but do you have. Do you have any questions about that so far?

B (26:18)

I do, yes. So I work with startups and I'M sure this question could apply to maybe companies that are really new to the cloud where this is their first time actually planning an active, passive active active architecture. So they might not necessarily know what the sequence should be. So are there, is there like maybe any pre made like templates, default recommendations for customers that are like well what does AWS recommend, what the sequence of steps to be if I'm not really sure what they should be?

C (27:01)

Yeah, I think Jeff is going to have a really good set of answers for that I think. But I'll tell you programmatically at launch we don't offer drag and drop template selection but that is something that we're looking to add in the future is to say I'm a startup, I have a standard three tier architecture, can you just tell me which plan I should use? And we're actually looking for ways to make that selection more dynamic in the future. But I'll say from my perspective, we have well architected framework is something that we offer through Amazon and that gives you a general overview of best practices for building in the cloud. But Jeff, you probably worked with more startups than me.

A (27:48)

Yeah. In addition to the well architected practices, we have some prescriptive guidance out there on our website where we actually walk people through specific infrastructures for, you know, different types of industries or different types of workloads. And with these new features you can very much expect that we'll be updating a lot of the technical content that's out there for application Recovered recovery controller to give some more guidance around, you know, what high level scenarios might look like. It is a relatively new release, so depending on when you're listening to this podcast, it might not, might not all be out there just yet. But that is something that we're very interested in getting updated and that is it'll be heavily driven by the resilience community.

B (28:39)

Yeah, I'm really excited for customers to start using this and it sounds like there's even just like a layer deeper of okay, you can have active, active, active passive, but even within those there's different sounds like levels of granularity of what that even means. So maybe Scott, could you elaborate? Like for example you were telling me that there's a concept of passive auto scaling.

C (29:05)

So yeah, so we, we have blocks that handle different phases of the recovery journal or journey. Sorry, I'm always going on a recovery journal, I need to start going on a recovery journey. And one of the first things that we would do is scaling. You know we have an EC2 auto scaling block ECS scaling and EKS at launch. And what this is is customers can set a percentage of scale that they would like to have in their target region, which is something I should mention real quick. We build these recovery tools and it's always interesting to build them because they have to be more available than the applications that run on top of them. You know, if the recovery tool isn't available during a live event, then, you know, what have we done? We've built something that broke. So we go to extra steps to make sure these are available. And what we did with regionswitch is we built a regional data plane, meaning that our data plane is in all commercial regions at launch. And the dependency posture that we've taken is very similar to a customer's recovery strategy, where we say if you've built in the west and you recover in the east, you've taken a dependency on the east. That's where our region switch data plane would live. So we would live in both sides of your recovery, your recovery journey scaling is the beginning of this. And through these blocks, customers can preemptively say, I would I need to be this scaled, or I need to be scaled above this percentage in my target region in order to operate successfully. These blocks do have a concept of passive evaluation, which runs every 30 minutes. And make sure that the resource configuration across your active region and your target region and IAM permissions, things like that are sane. They, they check out, they didn't break between runs. So. So that's one of the things that we do proactively with our scaling. We also have recovery blocks like actions that will actually shift DNS shift traffic away. We offer that through Arc Routing Controls, which is another application recovery service that helps you move traffic between regions. And we have a new execution block or a new concept here called Route 53 Health Checks. And what this is is a managed function that updates a Route 53 health check and redirects traffic based on your DNS configuration. So I think a lot of customers are using health checks along with DNS config to move traffic. We just kind of bundle that experience in an execution block. That's probably more than you wanted, Gillian. You just wanted to hear about passive.

B (32:09)

No, I think this is exactly the kind of stuff that gets like, people super pumped about multi region. And there was something that you were saying earlier. No, I think I forgot what it was, but you said something and it made me think, oh, I wanted to go there. Oh, okay. So, Scott, question for you. So Jeff was making a very great recommendation earlier about being able to test often their multi region strategy. And I'm curious if this new service region switches how that influences like their ability to test their multi region strategy.

C (32:56)

Yeah, it's a good question. And it's, I mean to be clear, it's always going to be challenging to test multi region recovery. It's a big action to take and if you're not able to confidently do it, it's something that people are afraid to do and there's a lot of finger crossing that things work during a live event. So we've tried to make that easier by consolidating all your recovery actions in a single service. And we've added a considerable amount of monitoring, both console facing and backend monitoring and logging so that you can see exactly how your recovery is going at each step of the process and then at the end get a consolidated report of how that recovery went, which we're making available as evidence for even regulatory requirements. So I guess what I'm trying to say is we've taken, we've taken a process that involves dozens of steps across dozens of teams and we've put that all in a single place. It still can be challenging and scary to test this out, but now there's a lot fewer threads that you need to chase down during that testing.

A (34:14)

I wanted to jump in real quick there. You made a great point about regulatory compliance changes in the past. Simply stating that you had designed a system to be resilient and redundant was sufficient for many, many regulatory areas. But we've seen a lot of shifts in modern framework to actually want evidence. Right, to need to see the details and to see that these things have been practiced and having that out output from region switch helps right away with hitting some of those regulatory requirements.

C (34:48)

That's the hope. Wow.

B (34:51)

So I've already learned a lot. I'm sure all of the listeners as well have really learned a lot and I'm sure they're super pumped to start using this. So Jeff, I'm curious. You've been working with probably hundreds by now of customers on their just like resiliency strategy. Any last piece of advice that you've had from based on the hundreds of conversations you've had that can help folks today?

A (35:19)

Yeah, I think it's important to understand and this is a little bit of a deeper topic so I'm going to brush it very quickly. It's important to understand where the different data plane and control plane actions are in your, in your infrastructure and to build applications that can fail over without dependencies on control plane operations. You want things to be running in the data plane. That means that it's already something that's standing up and functioning. You want your recovery efforts as much as possible to be things that can be executed through data plane changes. Without that heavy dependency on control plane and understanding where those boundaries are, it takes a deep dive into some of the documentation to really understand what that impact can be.

B (36:11)

Wow, I'm. It sounds like based on that suggestion that that's potentially a mistake that you've seen some customers make and it shows then the importance of really testing. Because I could imagine if you, even if it's a smaller application, you would really have no idea what would happen unless you actually test.

A (36:35)

Yeah, absolutely. Understanding what even, even things that seem simple, like you're changing, you know, a DNS pointer to, to move to a different region, understanding what that change looks like, how long it takes for, for that change to replicate through your systems you'll find things like client caching that maybe you hadn't considered. So really running those tests repeatedly and documenting the results and then going back and iterating and you're not going to solve every problem all at once, but as they come up, you know, capturing that information somewhere, prioritizing and, and moving forward until you've got a, a clean and repeatable process.

B (37:15)

I love it, Scott, so I'm curious for you if there's any like, advice that you have.

C (37:24)

Um, I mean, you know, it's a balance between I and I think Jeff actually said this earlier, the balance between cost and availability here. Um, I would say for testing specifically, it's going to be a lot easier to test an active active application than active passive. Just because by definition you're still active in one region and maybe you're balancing traffic there. And it does help you test the mechanisms of your shift, if not the full end to end active passive story. So I think it's really down to the, to the customer to try to see where that balance is. Like, are you happy just operating active passive and maybe testing once a quarter? Or do you want to spend a little more money with more infrastructure and testing more frequently? It's all going to depend on the individual needs of your customers.

B (38:23)

I love it. Really exciting. And last question for you, Scott. Where can customers go if they want to learn more about RegionalSwitch?

C (38:35)

Well, I would start at the Amazon Application Recovery Controller website. That's the landing page for all of our services. Region switch should pop right up from there.

B (38:48)

Awesome. And is there any place that you suggest customers go to if they want to maybe learn more, reach out to you on. On the Internet.

C (38:58)

I the learn more. You can't go wrong with well architected framework. I mean spend, spend time on that. That's constantly evolving, constantly growing. That's a pretty good record of our best practices. But beyond that, contact information's on the website. You're always welcome to reach out. We love hearing from people, especially with a new service like which parts are resonating, which parts aren't. We've got a pretty full roadmap for building onto this service. We'd like to have it informed by your needs. So please don't hesitate to reach out.

B (39:33)

Perfect. And Jeff, anything with you, Any place that you'd suggest if people wanted to reach out to you with any questions?

A (39:42)

Yeah, yeah. I mean, you know, Scott mentioned the well architected, the reliability pillar there is all about resilience practices and there's a lot of information, a lot of, a lot of guidance that is not specific to aws. Right. Well, architected is about how would you do this? You know, what are the things you need to consider regardless of where or how you're building. And of course we do have some suggestions that lean heavily on our tools. But well architected is really meant to be agnostic best practices. And regardless of where you're built, you're. If you have an account team, your account team can always reach out to, to specialists through, through many internal channels. We have ways to, to, you know, coordinate with, with our RSA community. If you want to have a, an executive briefing, those can be coordinated. We have resilience topics within all of those different things and of course we're just publishing content all the time under, mostly under architecture. You'll also find us at, at any local summit and we have a huge presence at Re Invent. So stop by the resilience booth or the Ask the Experts table at any of those events and we'd be happy to talk about anything that's on your mind.

B (41:03)

Great call outs. Well, Scott, Jeff, thank you so much for being here today on the AWS podcast.

C (41:11)

Thanks, Gillian.

A (41:13)

Yeah, thanks for having us.