← AWS Podcast
AWS Podcast
#689: Diving Deep into AWS Transit Gateway
00:28:31
In this episode, Simon Elisha and Brett Looney dive deep into the AWS Transit Gateway, a cloud-scale
Loading summary
Transcript98 lines
- [00:00]
A
This is episode 689 of the AWS podcast, released on October 14, 2024.
- [00:09]
B
Hello everyone, and welcome back to the AWS Podcast. Simon here with you. Great to have you back. I'm joined by my special guest, return guest Brett Looney, who is a principal solution architect here at aws. Welcome to the podcast, Brett.
- [00:19]
A
Thank you for having me back, particularly so short after the last.
- [00:22]
B
I know it's good we're getting on a roll here. So, as I think I may have mentioned in the previous podcast, Brett knows more about, or actually Brett has forgotten more about networking. So he is my and many other people's go to when it comes to networking stuff in general, but also on aws. And today we're going to dive deep into something pretty cool. And this is something called Transit Gateway. And it's actually very advantageous, I think, to revisit the current state of the art of cloud networking, because it changes all the time. This is the thing. This is not the days of get your ccie, wait three years, get another one. Things may have changed, a new generation may have come. This stuff changes all, all the time. So, Brett, you're going to help us, aren't you? You're going to help us get through this.
- [01:09]
A
And you've just triggered me with the CCIE stuff.
- [01:12]
B
Sorry. We both used to work for Cisco, so we have those memories. Let's start with the basics. What is Transit Gateway and how does it relate to VPCs?
- [01:21]
A
So the big thing I want everyone to take away from this is that Transit Gateway behaves like a cloud scale router. As in you connect your VPCs together and they can all talk to each other. At its simplest, that's basically it under the hood, of course, which is where we'll get to. It does a lot of other things think router, but it's not actually a physical box sitting somewhere.
- [01:45]
B
Yeah. So don't think of the constraints of a physical box that you would have thought of in the past. Think of it as a highly distributed router.
- [01:52]
A
Yes, yes, exactly. That is.
- [01:53]
B
And what's it connecting? If we're routing, what are we connecting to? What kinds of things can we connect to?
- [01:58]
A
So the idea is, at the beginning it's VPCs, but in reality it is everything else in the networking space that AWS has. So your VPCs can talk to each other. If you have direct connect, you can bring it into your Transit Gateway. It's just another point where you connect to the router. Simon can see me. I've got air quotes happening yeah, there's the air quote.
- [02:19]
B
You need your air quote voice.
- [02:21]
A
Yeah, right. But also VPNs and also a very special thing off to the side called Transit Gateway Connect that lets you bring in your SD software defined WAN into it as well. So it becomes the central point where you route all the traffic inside aws. Okay.
- [02:38]
B
So it's the place to be when it comes to routing a traffic. So probably good to know about it. But the question often comes, well, why would I use that? Rather VPC peering. You know, we introduced VPC peering a while ago. I know how to use it.
- [02:50]
A
Absolutely.
- [02:50]
B
Why change?
- [02:51]
A
And VPC peering is very simple, right? You peer two VPCs together, they can talk. Awesome. You have three VPCs. Great. You've got A, B and C. As long as you peer them all together, they can talk to each other. VPC peering is non transitive though. If you have A peer to B, peer to C, then A and C can't talk to each other unless you peer them. And this is where the problem comes. As your organization grows and you get to 10 VPCs, 20 VPCs, 100 VPCs, now you've got to peer all the VPCs together that you want to talk to each other, which becomes painful, it becomes hard to manage. And on top of that there's a hard limit of 125vpc peers per vpc. If you bump into that, that's a problem. And of course most organizations eventually get to the stage where they go to I've got 10.
- [03:41]
B
It's going to take long. Yeah, that's not going to take long.
- [03:43]
A
And it's hard. So what do I do? And Transit Gateway allows you to do one connection per VPC and you're done. And also now I can share my Direct Connect and my VPN with all of my VPCs. So for any non trivial organization it makes sense. But when you're just starting, sure. VPC peering is awesome. It's great.
- [04:02]
B
Yeah, it could be, could be useful for. Look, there was at least there used to be little things to watch out for with peering. Like you couldn't overlap IP address ranges and other stuff. And that may have changed in the interim. I haven't been keeping track of that. No, you okay, okay. I'm still up to that. I still get my network certification, but yeah. So there's edges you can run into which with Transit Gateway you don't have to worry about. And I think you talked about that hard limit of VPC peers. What kind of limits are we talking about for Transit Gateway So the base.
- [04:30]
A
Limit, and this is a soft limit for Transit Gateway for attachments, and I'll talk about what an attachment is in a minute, is 5,000. So if you want to take 5,000 VPCs and a couple of directing exits and VPNs, no problems, you don't have.
- [04:43]
B
To talk to anyone about that. You can just go ahead and do it.
- [04:46]
A
Absolutely, absolutely.
- [04:48]
B
Nice.
- [04:48]
A
Right?
- [04:50]
B
But it costs, costs us. We have to spend money to make this happen.
- [04:53]
A
Right, right. So you are paying for a service that is routing packets. The cool thing about VPC peering is that when you peer the two VPCs together, if you have an instance in VPC A and an instance in VPC B, the VPC overlay network does some magic and it just sends the traffic between those two instances. There's actually no router involved. It's so cool from just a perspective to see that stuff happening. But therein lies the limit in that all the hypervisors have to know about all the VPCs that appeared and that's where that limit comes from. And so with Transit Gateway, you don't get that limit. But now we've got to maintain this highly scalable distributed magic cloud router. And that actually does cost money. So I actually have customers where they say, actually We've got these two applications in VPCs that are connected via Transit Gateway. We have hundreds of VPCs, but these two particular applications are very chatty. We're talking, you know, terabytes of traffic per hour. How do we reduce the costs? And you know what, you can use both of these things at the same time. So they use Transit Gateway for all of their regular traffic. And these two VPCs that have a lot of traffic between them, they just peer them together immediately.
- [06:04]
B
It makes sense to do so for that.
- [06:06]
A
Yeah, right. And it's a little more complicated, but it's not crazy complicated as in peering all the VPCs and they get a huge cost saving. So it's not a one or the other, but most places go for Transit.
- [06:17]
B
Gateway because I think it's the right starting point, at least, unless you've got an exceptional situation. What about bandwidth? What are we looking at from a performance perspective? Just to help us design at a.
- [06:27]
A
Performance level, it starts at 100 gigabits per second per availability run. So there's lots of bandwidth there. You shouldn't run into any limits. If you do come chat to us, we're always interested in people pushing the limits of the services.
- [06:41]
B
Great. Now let's talk. You touched on the concept of attachments let's dive a little deeper and understand from a mental model perspective, from a structural perspective, how we think about these things and how we hook them all up.
- [06:54]
A
Cool. So if you're a router person, you'll know about router interfaces. They're the physical ports that you plug the stuff into on the router. In the transit gateway world, an attachment is a routing interface. I attach a vpc, I attach a direct connect, I attach a vpn. From the router's perspective, they're all just interfaces. I send packets to them, they all behave exactly the same. It's only at the human level where we'd look at it and go, oh, actually that goes back to on premises via direct connect. This goes to a vpc, but they're just attachments, they are just router interfaces.
- [07:29]
B
Okay, well that makes it pretty easy to just understand how we move through things. But how do we manage the routes? How do we know where things go? Route table management can be fun, particularly when you get it wrong.
- [07:43]
A
Fun is always a euphemistic word, right?
- [07:47]
B
It's fun with a capital ph.
- [07:51]
A
So a route table inside transit gateway controls where traffic goes, when traffic comes from an attachment. So if you have a route table that is associated with an attachment, those are the rules in that route table that we use to send traffic wherever it needs to go. The simplest transit gateway design has one route table, and the one route table says all the traffic can just talk to all the VPPs, everything. And that's the simplest way to go, which is great. But often you might want to do things that are more complex than that. So you can create multiple route tables. You can say, actually, I have dev and I have prod. They have two separate route tables. Dev can talk to dev, prod can talk to prod. Very simple. Two route tables makes sense. But you could also have a thing where you say, well, I've actually got the shared services vpc, you know, with things like active directory and stuff like that, I want everyone to be able to talk to shared services. But outside of that, prod is separate to dev. And so your route tables give you that level of control to do that.
- [08:56]
B
And do I have to define them statically or can they, can they learn over time? Can they figure things out? Can I teach them?
- [09:03]
A
So that is the third interesting word you'll find in the transit gateway documentation. The console is propagation. And propagation is how does the route table learn the route? And just like a router, they can be static, but they can also learn from the attachments. So from if you're propagating a route from VPCA into the route table, it will automatically learn the routes in VPC A. If you're propagating the routes from your direct connect, then we learn the routes from bgp, which is the routing protocol that we run on the direct connect. And so therefore you can have a fairly dynamic network, but you can still have fine grained control with static routing.
- [09:44]
B
Gotcha, gotcha. So you can mix and match dynamic and static routing. For those who may be newer to it or just have forgotten over time because it kind of gets buried there. BGP is kind of really important border gateway protocol. So, you know, help us appreciate the majesty of BGP and why it's important, why things, if things ever go wrong with it, everything stops.
- [10:07]
A
BGP is the protocol that is the backbone of the Internet. It's designed to do large scale route distribution. It also has lots of little knobs and things that you can turn in there to say, well, I've actually got two paths or three paths and I wish to prefer these paths over the other. In a corporate environment like a traditional enterprise network, you probably run OSPF or if you're a Cisco person, eigrp. There you go, I've probably just triggered you. And there's a couple other routing protocols. But when it comes to connections between disparate organizations, BGP is the choice to do this. And BGP is a little bit of a learning curve if you haven't done it before. If you're just setting up a single direct connect, it's relatively straightforward, but it does get a little more complex when you go, actually I've got a direct connect with a backup VPN or a direct connect with that backup direct connect. I need to do a little bit of route manipulation to go, this is my active and this is my standby, or I want both of them active at the same time. And BTP gives you the flexibility to do that.
- [11:11]
B
Yeah, it makes, that makes that work. It is an amazing thing because it really does. And often you'll see if you're watching media and you see bad things happening at a global scale or network level, it's often someone published the wrong BGP routes and, and everything went real bad.
- [11:29]
A
Right. And there's actually some work going into the BGP protocol now to make that more secure, as in the ability to cryptographically sign the routes that you advertise so that nobody else can hijack them, which is interesting. Trying to retrofit that on, you know, hundreds of thousands or probably Millions of.
- [11:43]
B
Routers across the world, how hard could it be? Yeah, let's talk about another concept because, you know, life is important to have lots of three letter acronyms in it. So we're going to talk about vrf. What is VRF and how does that fit in here?
- [11:57]
A
So VRF is virtual routing and forwarding. And in a carrier network you would use it to have customer A, customer B, customer C, and keep them separate to each other. In Transit Gateway, you get the same capabilities using the route tables that we were talking about just before. In that you can say I want to separate my dev and my prod and my UAT traffic, or I want them to mix in certain ways. So without using the term VRF as a whole, I would like to call it VRF Lite. It gives you the ability to maintain traffic separation if you want to. And in many organizations that's exactly what they want to do. But to our previous podcast topic as well, which was about how do we filter traffic and inspect it on the network? It also gives you that power to go, I want to separate these. Unless the traffic goes via this firewall over here in my inspection vpc, I can use the power of the route tables to maintain separation, except in very, very controlled circumstances, which keeps security people happy, it keeps the risk people happy. And it's exactly the same sort of model that you would have on premises by putting a firewall in between your dev and your prod networks. It's the same model.
- [13:06]
B
Now it's interesting you talk about that like a lot of the concepts are talked about of, you know, it's like a router, it's like a familiar construct. But this is, I emphasize, this is not a honkin great device that you can poke and prod and fiddle with. It's way more special than that. Help us understand what makes it different, what makes it work in the cloud, quite frankly.
- [13:28]
A
Yeah. So I'm going to throw a bunch of terms at you. Right? So it's a cell based, random shuffle, sharded distributed routing engine that explains everything there, that also practices the concept of constant work. Now, so this is buzzword.
- [13:49]
B
There's a lot we're going to unpack here. This is why we take the time to do it.
- [13:53]
A
Right. And so the underlying service that runs Transit Gateway is called Hyperplane and it's actually the thing that has all of those various concepts embedded in it. And if I was to do a huge disservice to the Hyperplane team, I would call it a magic packet rewrite engine.
- [14:11]
B
I think they'd be quite thrilled to hear it just going that way.
- [14:14]
A
And it's incredible because it not only powers Transit Gateway, but some other services as well. But the idea is that Transit Gateway team gives some code to the Hyperplane team and they say, please run this for us. And it goes away and runs it. And so each customer gets their own cell. You could think of it like a Docker container. Like it's a boundary within which the code runs and it can't escape from. And it's given a specific job to do, in this case, route packet. The cool thing about the cells, they can scale up and down to meet those huge bandwidth requirements, but also without wasting resources. And because they're cells, we can pack them closely together onto a level of compute. So under the hood, right? It's just compute, routing packets, distributed compute.
- [15:05]
B
But it's the magic of how it's organized to work in that way.
- [15:09]
A
Right. So then we make sure that your cell has multiple sets to it. There's at least one in every Availability zone. Okay. Because we need that. Because what we don't want to be doing is taking the traffic out of one Availability Zone and sending it over to a big router, honking router, and the other Availability Zone and bring it back. That doesn't make any sense. So the cells run independently in each Availability Zone. And then in order to avoid a whole bunch of problems, including mass hardware failure that affects all the same customers, but also to avoid noisy neighbor effects, right. We do this thing called random shuffle sharding. So if you've got customers A, B and C who are running their cells in Availability zone A, in Availability Zone B, we make sure that they are not in the same mix. So A doesn't share compute with B or C. And that way, if one of them is a noisy neighbor, the limit to that is very, very small blast radius. There's a great article, by the way, on the Amazon Builders Library, which talks about Route 53 and shuffle sharding and how we do exactly the same thing, but it's an identical concept.
- [16:25]
B
Yeah, the concept is very consistent.
- [16:28]
A
It's way cool. And we do this randomly, and if it's not random enough, we have an algorithm that moves things around. And this also helps us avoid hardware failures. If we detect that a failure is about to happen, like we know there are memory errors or the hardware errors on the underlying compute, we can move people around to other hardware without them noticing. And so it's lovely, right? And I'm like, again, this hyperplane Thing is magic.
- [16:51]
B
Yeah. It's unlocking so many different things that you can't see. And that's the idea that you can't see. It's good that you can't see, but my goodness, it's nice to know it's there.
- [17:00]
A
Exactly, exactly. So the final thing we do is the configuration for each of these cells is delivered to the cell. It's actually read from S3 and the cell reads the configuration and it goes around, does stuff. What we want to make sure is that if we send the cell some huge configuration that it's going to be able to process it. So this is the theory of constant work. We always send a full configuration file to the cell, even if the configuration file, some of the instructions are, you don't need to do anything here. And therefore we're assured that if we have a customer that comes in with a really, really large routing table that we already know it's going to handle it because every single time the cell loads its configuration file, it's loading the full configuration file, not a really short one. And this is just a way of making sure that everything runs very, very reliably.
- [17:53]
B
What's this? It's that consistency and it's an interesting pattern and it's worth just touching on a little bit more because it's. It's kind of counterintuitive to an optimization mentality of, well, I'm only going to send the exactly the bytes I need at any given time and I'm going to save the world because I'm doing that. There's a place for that and it's appropriate in certain situations. But this is a great example. It's actually not the right way to go counterintuitively.
- [18:14]
A
Yes, exactly. We want to prove that the software is fully exercised all the time and that's the best way to find those.
- [18:22]
B
It's a really interesting mental model because it's almost like by working hard all the time. It's not working hard like it's just what it does is doing its thing. Now you touched on Hyperplane a few times and you've given it the moniker, I think of the magical. What else is it used in? Clearly this is a very useful technology. Where else does it pop up in the networking world for us?
- [18:43]
A
So it's actually behind a bunch of other services. So Network Load Balancer is a hyperplane server, Privatelink is a hyperplane service and Private Link in itself does different stuff. If you think about the way Private Link works, it is used for software as a service providers or for AWS services to basically give you a private interface in your vpc. And its big ticket is it does double sided nat. So that's network address translation. Sorry for the non networking people where we don't care what the IP addresses are on either side, you can just talk to each other. And again, this shows off the power of hyperplane because the code that runs Private Link is different to the code that runs Transit Gateway and yet it's using the same underlying architecture to run. So again that same cell based shuffle.
- [19:29]
B
Sharded that concept filter through.
- [19:31]
A
Right. It also runs Gateway, load balancer and NAT Gateway. And of course there's a few other ones as well. There's about six or seven. But the intention is that we've got this amazing service that scales horizontally to give us huge amounts of performance and we can get lots and lots of services running on it. And again at huge scale, which is so good.
- [19:54]
B
And this horizontal scalability I think is something we need to remind ourselves of because Certainly back in 2011 I talked about horizontal scalability a lot and it was new for people and we used to big vertical boxes but sort of forgotten about the magic of it. But none of this happens without massive horizontal scalability. Like you could never build a box big enough to do all the stuff that all customers want to do. And so being able to scale horizontally is the magic quote unquote. That makes it all possible and also avoids things like, you know, my firewall box is too small now and it's throttling my customers, et cetera. That's exactly what we don't want to have happen.
- [20:31]
A
Exactly, exactly. So things scale for you and you as a customer don't have to think about it again, you're plugging in this router interface so the attachment and that's it, you're done. You don't have to worry about it anymore. And again, it does come. And we did talk about cost before, you only pay for the traffic that goes across it. Yes.
- [20:51]
B
So still pay as you go. Pay for what you use. That hasn't changed.
- [20:54]
A
Yep. So there's an hourly charge for the attachment to the vpc, but you're only paying for the traffic that you send. So if there's lots of traffic, you do that again, you don't have to then worry about sizing for peak, as in I need, I need, do need this really, really big router because once a month I do have to send 100 gigabits of traffic for an hour. You don't need that.
- [21:13]
B
Yeah. Yeah. What about resilience and redundancy? Like, how do I think of it? Do I need, do I need multiple gateways? How do I structure myself?
- [21:21]
A
Yeah, and you don't. Because this, again, this horizontally scaled shuffle sharded cell based application is taking care of that for you. And indeed, even if you did say, okay, I want a. And Transit Gateway, because I just want to be sure it's going to run on the same infrastructure so it actually doesn't help you and in fact it complicates your life more by doing that. So again, I know it's a big ask for our customers to trust us, but it's also the reason that I'm talking to you today. Right. Which is the architecture and the way we design things is designed not so that there are never failures.
- [21:53]
B
Everything fails all the time, as we say, but it's designed to understand that factor.
- [21:58]
A
Right. It's designed to be as resilient as possible all the time. And again, even, and I know this from, you know, my work back in the industry before coming to aws, things failed and when your single large router failed and you had your standby router sitting there, it took seconds to kick across. And that's. And that's our aim to customers as well.
- [22:17]
B
Yeah, exactly. I remember with those core router failures, etcetera, not only did it take a little bit of time for it to kick across, but there was always the fingers crossed moment of. Are the configs the same? Right.
- [22:27]
A
Am I running the same software on both of them?
- [22:29]
B
Yep. Yep. And sometimes you don't want to run the same software on both of them because both of them have the same bug.
- [22:35]
A
Exactly, Exactly.
- [22:36]
B
Running infrastructure is hard.
- [22:38]
A
It is, it is difficult. And, but this is, this is why I love this. Right. The VPC construct takes away the running of the base network from people. Transit Gateway gives you that core router functionality which you expect and you expect the reliability of it and you expect it to scale and it does those things in its sleep.
- [22:58]
B
So then, if I'm hearing this correctly, if I'm contemplating an environment of any degree of size, so beyond one VPC, one account or a couple of VPCs, and that's all it's ever going to be. I should be starting with Transit Gateway and I should be using its features and functions to make my networking life a lot easier.
- [23:16]
A
Absolutely, absolutely. And it's interesting you mentioned cross account there as well. Transit Gateway doesn't care what accounts the VPCs and the direct connects and the VPNs are in.
- [23:26]
B
There's another benefit there.
- [23:28]
A
And again, it actually gives you the flexibility to say, you know, I have a networking team on premises that takes care of things. I want a networking account in AWS where my transit gateway and my routers live, but I want my VPCs to live where the applications are, which is totally fine. It's completely cross account aware if you like, and lets you do.
- [23:48]
B
And what about across regions? How does it work from a multi region perspective? How do I think about it from that side?
- [23:53]
A
Yeah. So transit gateway is a regional construct in the same way that VPC is. If you're operating in say AP Southeast too, that's Sydney for the uninitiated. You have a transit gateway there, you have your VPCs attached to it. If you now want to operate in another region, AP Southeast 4, hello Melbourne. You put a transit gateway in that region as well. And then you can peer the two transit gateways together. And guess what the other transit gateway.
- [24:20]
B
Looks like an attachment, a router, it's all attachments.
- [24:24]
A
So it's like plugging two routers together across that. Right. But that also gives you. It's a good thing because now your traffic that is in Melbourne stays in Melbourne until it needs to go to Sydney and vice versa. So you get that local routing capability.
- [24:37]
B
Fantastic. So where does someone start? Like do they just hop in the console? Do they do a tutorial? Do they God help me, read the documentation. I mean, where does one start? And let's actually, let's pick two Personas. Let's pick the. I'm a grizzled network administrator. I've seen it all, been there, done that. And then let's pick. I've just graduated from university and I have no idea what I'm doing, but.
- [25:01]
A
I'm really excited for the grizzled network administrator with the battle scars to prove it, AKA you or you. I would jump in with both feet, create a transit gateway, attach some VPCs to it, do some routing. After that we do have workshops which are very much okay, what happens now? If we want to separate out those VPCs again, dev and prod and make they don't talk to each other, let's label with multiple route tables, do that sort of thing. So again, from there it becomes more and more advanced for the newbie and that's not a bad thing. We're all newbies all the time.
- [25:41]
B
It's nice to be often. It's fortunate to not have to unlearn stuff. So let's celebrate that.
- [25:46]
A
Exactly. I would actually first just start with it with a vpc because interestingly the concept of route tables actually exists inside of VPC as well, which is hey, I've got like five subnets, how do I make them talk to each other? The default is I've got a single route table but actually I want to separate the subnets so some of them can't talk to each other. If you can get that concept, then you're ready to go and be the grizzled network administrator and play with Transit Gateway. Again the workshops are very handy, but I would stress stay on the simple side of it, which is I have three VPCs, how can I make them talk to each other and start with a single route table and make sure the traffic flows. There is a great service here by the way, called the VPC Reachability Analyzer. And you can go to the VPC Reachability Analyzer and go, can instance A talk to instance B across Transit Gateway? And it looks at security groups and KNCLs and the route tables and it evaluates it without sending a single packet and says yes they can or no they can't.
- [26:46]
B
And this is why it uses maths, automated reasoning.
- [26:50]
A
It's beautiful thing, it's absolutely incredible and in fact, sorry, we're, I'm going up down a rabbit hole because this is such a, such a cool topic. I've actually recommended to customers that they use it because you can use the VPC Reachability Analyzer on an ongoing basis. As in you can automate it to go. I want to make sure that after a deployment, instance A can always talk to instance B. And if you don't, you can then trigger a thing and roll back your.
- [27:15]
B
Deployment, whatever and you've got the information as to why it's not exactly.
- [27:18]
A
Not talking to one another exactly, but you can do the opposite as well. You can say, actually I want to make sure that instance A can never talk to instance. Yes, like that is verboten. And so again if I make a change and that makes it possible, then I have something to start with. So very, very powerful tool. And as you say, it uses math, math and reasoning and logic to do its work.
- [27:36]
B
That stuff well. And also it sounds to me like the best friend of the grizzled network administrator because it sounds a lot more fun than crawling around in a data center, unplugging, replugging, using wireshark, figuring things out, changing a firewall, setting up the process of elimination in highly air conditioned circumstances is not the way you want to spend your Friday night. Not that that's ever happened to me.
- [27:59]
A
No, no, no. And then you find yourself locked out of the room you need to be in.
- [28:02]
B
But that's now we're getting really bad. Brett, thanks so much for coming on the show and demystifying Transit Gateway for us and continuing our journey into the networking side of things.
- [28:13]
A
My pleasure. I'm always happy to talk about these things. And again, customers who have questions should reach out to their local FA and we can again demystify it in Perth.
- [28:22]
B
Absolutely. And you can also reach out to us directly. AWSpodcast@Amazon.com is the place to do it. And until next time, keep on building.