← Sharp Tech with Ben Thompson
Sharp Tech with Ben Thompson
(Preview) AWS History and Trainium’s AI Future, OpenAI Makes a Deal With Microsoft, Meta and the Future of Wearable Devices
00:24:02
Loading summary
Transcript47 lines
- [00:00]
A
Hello, and welcome to a free preview of Sharp Tech. Hello, and welcome back to another episode of Sharp Tech. I'm Andrew Sharp, and on the other line, Ben Thompson. Ben, how you doing?
- [00:19]
B
Perturbed. Andrew. I have a new computer, so it's complicated, but all the recordings happening on my main computer, but. But I need a computer in front of me for the rundown, which may or may not be accurate. You sometimes just suddenly pull out things that are not there. And I'm like, what the happen?
- [00:37]
A
I gotta keep you on your toes.
- [00:38]
B
And you're like, oh, sorry, I'm reading out of. Yeah. You're like, oh, I made my draft in Gmail, which I'm reading out of. I'm like, I'm using Microsoft Word for your sake. It's unbelievable. But so I have my. So what happened was I have had. I think I've talked about this. I have two computers. I have my MacBook Pro that mostly stays on my desk. If I'm traveling and working, I will bring it with me. But it is basically a desktop computer, and it's great. The MacBook Air is amazing for sort of carrying around having with you at an M2 MacBook Air for a long time. Love it. Amazing computer. Unfortunately, because I carry it around all the time. It is like the baseball games and practices and I'm working in the car. It's gotten dropped. At one time, it was left in the backseat of the car. There were two boys in there. It's like, all stomped on. It's got, you know, but that's fine. I mean, it's not fine, but, you know, it's okay. There's nothing important on there. The problem is it started Colonel panicking. So last week, I think we talked about this, my computer crashed. The crashes are accelerating in frequency.
- [01:40]
A
Oh, interesting. Okay.
- [01:41]
B
Well, what happens is, especially if you drop it, stuff inside gets loose and it starts sort of, like, shorting out. And like, that's like an unrecoverable error. The whole thing is just going to, like, go. So I was pretty sure. But yesterday it got really bad. Like, I need a computer for the podcast recording tomorrow. So went to the Apple Store, just bought a new MacBook Air.
- [02:01]
A
Okay.
- [02:02]
B
The problem is the MacBook Air runs Tahoe, the new Mac OS. And everyone's been complaining about Tahoe. And I validate most of those complaints.
- [02:13]
A
Yeah.
- [02:14]
B
But there's a complaint no one's mentioned, which is I kept feeling like it was weirdly dim, and I'm like, what's like, I checked the specs. Did they change the screen from the M2 MacBook, of course. No, they didn't make a dimmer screen. What turns out my. My Tahoe complaint. I just want to add it to the list of everyone else. The interface is so white. It's white on white on white on white everywhere, that it makes the whole thing seem dim. It's like it's. It's a really bizarre effect. Yes, I could do dark Mode. Unfortunately, I'm 45 years old. We're circle back to me being 45 years old. My eyes set at the end. No dark mode for me. So anyhow, I just want to register. Add to the Whitney of Tahoe complaints. Too much white contrast, please.
- [02:58]
A
White. Well, I have a reveal live on the podcast. I too, got a new computer this week because my computer, I have a MacBook Pro that I keep on my desk at all times. That's my podcasting machine, and I use a MacBook Air all over my house. It's what I write on, it's what I prep for shows on. I'm using it on every floor of my house. And when you have two kids running around, they tend to pick it up and drop it and do all kinds of unhelpful things with your MacBook Air. So I had to hide my computer by my bed one night, and then I wound up stepping on that computer as I got out of bed. And so it just completely shorted out. Out of nowhere again, some screws probably broke loose, and then a day or two later, it just stopped working entirely. So I got a MacBook Air. And I will say I feel the software pain with my new MacBook Air. Like, there are so many things that I've had to resort to ChatGPT for to try to fix on my MacBook, which should not be how these machines work, right?
- [04:12]
B
They should be getting, like, nicer to use over time, and they're.
- [04:14]
A
They're more complicated, more frustrating. It is what it is, still a great machine and hopefully won't be a broken machine over the next couple of years because I'm excited about the new chips and I do love the air.
- [04:29]
B
In any event, consider getting. Did you consider getting a MacBook Neo? Because I'm sure someone is going to ask.
- [04:34]
A
No, I did not consider getting a MacBook Neo because I'm working on this, like, every single day of my life and will be for years to come. So I feel like it's worth paying the premium. I didn't go for the 24 gigs of Unified Memory. I was content with 16 gigs of Unified Memory. Although I felt kind of lame. Did you go 24 there?
- [04:57]
B
No, I got the base model 512, the absolute base model. Cheapest chip, least memory. Again. The good thing is this is actually very important. There was nothing important on my old computer, which was good because actually it was getting so bad by the end. It just really accelerated the last day. It took me like three tries to erase it because it kept turtle panicking before I could erase everything. So if I wanted to finish line. Yeah, it was rough, but no, this is like a netbook. Everything's online, it's totally disposable. I like the little stuff, like the light up keys. One of my biggest use cases is my son's at baseball practice, which is a long ways away. And I sit in the car and I work and so like having light up keys is actually useful.
- [05:45]
A
It's a light. Yeah.
- [05:46]
B
And like the ambient light adjustment. Like I love that feature. Neo doesn't have it, but I actually had to turn that off because the whiteness of the interface meant every change in brightness of the screen was felt like it was like putting a black pane of glass over the whole thing. It was just so dramatic to sort of shift. So maybe I couldn't fix the whiteness.
- [06:10]
A
Yeah, bring Forstall back and he could fix the whiteness for you. But a couple of new computer buddies here on the show today. Very exciting stuff. Stuff. Yes. Well, we are not going to be talking new laptops. We are going to be talking Amazon and OpenAI on today's episode. But before we get to Amazon and OpenAI, Amazon did release its earnings later in the week along with the rest of big tech. We're going to take some of that.
- [06:38]
B
Yeah, terrible. So you have Amazon, Google, Meta and Microsoft all in the same day along several other companies that I'm interested in. But I. This is the first time I've had one of these days where everyone's there. And also I thought like, oh, this interview running on Tuesday is gonna be great. Cause then I could at least hit one of the earnings Wednesday night for Thursday. And I forgot being in the US like the transcripts of these don't come out. So I wrote about Amazon in my update on Thursday in part because I've been talking about Amazon for the last couple weeks. So that made sense. And also their Transcript dropped at 9.01pm so it was the first one to drop. I think Google came out like 11 and then like I don't know. So I haven't read the Meta or Microsoft ones. I glanced at the Google one but yeah, that's we're going to be focusing mostly on Amazon both for topical reasons and also I can't stay up till four in the morning waiting for transcripts.
- [07:35]
A
Well yes, I look forward to immersing ourselves in the Meta earnings.
- [07:40]
B
The Google earnings, yeah, Taiwan, they were always there. It was great but another time zone advantage of being in Asia. But what are you going to do by the way?
- [07:47]
A
Is that normal for all of those companies to release earnings on the same day? Because I remember it being more staggered
- [07:54]
B
but yeah, no, they, they, they're generally all bunched together. I don't actually maybe so can email us. I don't actually know how that works, how and when they announce usually the announcement of the date is usually like seven to 10 days before it seems okay and obviously like that's a very Earnings has always been like a core thing for strategy and so it's something I'm always aware of and like thinking about as far as scheduling and you see this, not only were they all in one day, but also lately they've always been on Wednesdays. And then I'm like, it's very frustrating for my policy.
- [08:31]
A
It's not bad though from a strategic editorial standpoint. It gives you a couple days to sort of mull what you're hearing and
- [08:38]
B
then yeah, but I don't know, like Monday, I mean every time there's a meta 10% drop I'm like oh, this is my sweet spot. I want to get on it right away.
- [08:45]
A
But well, good news, the sweet spot will be waiting for you on Monday because Meta's taking some hits. For now though, we will talk about Amazon and I'll read from the Wall Street Journal. Amazon said Wednesday that its edge in cloud computing and aggressive investment in new data centers is translating into a surge in its artificial intelligence business. Chief executive Andy Jassy said that revenue from the company's Amazon web services grew 28%, the fastest pace since 2022, in part because many customers building new AI agents want them stored in the same spot where they maintain their other cloud services and data. Revenue for the period rose 17% to $181.5 billion, while net profit increased a sharp 77% to $30.3 billion, which Amazon attributed to pre tax income from its investment in Anthropic. Both figures beat analyst estimates shares were up more than 4% in after hours trading. So Ben, big picture, what is the story with these earnings? Has Trainium risen from the dead? Does the US AI story look better today than it did 12 months ago.
- [10:03]
B
Yeah, I mean Amazon is I actually think ended up down so far today. So who knows what's going on there. But the I thought, I've been thinking about Amazon. They are certainly one of the many participants in the ongoing AI soap opera. Who's up, who's down, everyone has their time. And I think there was a lot of concern, I think really crystallized in a semi analysis article a couple years ago talking about how Amazon is like screwed in for AI in the long run because of they won so hard as it were in the data center. And I wrote about this a bit. I talked about Amazon and Apple together, the two big winners from the cloud. Can they actually adjust for the AI era if it requires sort of different approaches? And the framing in that somebody else article was well, Amazon has invested heavily, not just of course they were first but but in really maximizing their position in a commodity market. And what I mean by that is we've talked about this on this podcast. There's two ways to make money. What everyone tech thinks about is make something that's differentiated, that has a moat and sell it for a premium. That is the Apple model. And everyone I was at some sort of startup event a couple weeks ago and they're just talking about moats, moats, moats. Like that's what everyone in Silicon Valley is like thinks about. It's the most intuitive I think but
- [11:28]
A
it's also what you're doing.
- [11:29]
B
Well the real. Yeah exactly. The real world where a lot of comp. Like a lot of products are commodities, they are highly substitutable. You can, if you don't get something there, you can get it somewhere else. How do you make money in that world? In that world you make money by having a superior cost structure. You can deliver the commodity at a lower price than everyone else. And the price for your commodity is not based on your cost structure, it's on based. You can always out compete everyone by offering a lower price. But if assuming there's sufficient demand, the market clearing price is, you know, if you have a perfect balance, assume you like demand and supply or can scale perfectly, the market clearing price is going to be the marginal cost to produce the commodity for the highest cost provider. So if it, if making a widget and the widgets are widely available, the price will be Whoever it costs company XYZ $10 to make the widget, then all the widgets in the world are sold for $10. Now again there's lots of variables here. Supply and Demand vary, there's elasticity on the price, how many people want to buy. But just if you're looking at that segment, generally for a commodity where it's totally substitutable across companies, the market clearing price is going to be the cost structure for the marginal cost of the most expensive provider. Right. So if you have four providers, it costs Provider 1 $10 to make it, Provider 2 $9 to make it, Provider 3 $8 to make it, and Provider 4 $7 to make it. The company that has sustainable profits in the long run is the $7 company four. That's right, yeah. They're making $3 of profit on every widget. And that's, that's how you could make a lot of money that way, like because everyone needs the commodity. And then you're also in a strong competitive position because you can lower the price. The price goes down, Company one will go out of business and then suddenly, then supply will go down and the price will go back up. Like, so you're just in a very. The lower your cost structure in the industry, the better you're at and the low, if you're at the lowest end, you have a lot of power and can be very, very sustainably profitable. So that was Amazon in the cloud computing age. It's not just that they were first to build out the cloud, but they were first to really invest in a few different things. So one was around, obviously their own processors. So they make Graviton processors. The number one use case for Graviton processors, which you could go as a customer and get an instance. Graviton processors, particularly in the early years, they sucked, so no one would want to do that. But Amazon didn't just offer infrastructure. You can go buy a processor, they offered platforms. So you can get the redshift, the Amazon database service. Right. And in this case, you're not actually, you don't know what the processor is. You're just getting database as a service from Amazon. And guess what? Amazon's probably powering Redshift with Graviton processors. Like, so their, their cost to serve it is diminished because they're using much cheaper processors and Graviton's gotten better and better over time. So. But because they can abstract that away with, especially with their platform stuff, they can have a sustainable cost advantage there. They also did a lot with networking and they built this entire co. Actually, the first, probably most important chip was this CO processor where with a server you have like a virtual machine, you have to actually run the actual server. And then on top of that you have all these virtual machines that appear as a computer to the client, but actually one chip, one computer is servicing hundreds or thousands of clients that are actually all sitting on top of it on their own little virtual machine. You need to actually run the computer though, and you need what's called a hypervisor to actually manage all those virtual machines. Amazon took all that off the main chip and had a side chip that basically handled all the networking and handled all the management of the system so that the big intel chips that power. That's why you had these intel chips with tons and tons of cores, because you could, because all those one core could be dedicated to a particular virtual machine. You could keep all that intel expensive intel chip capacity to run more hypervisor, run more virtual machines because you offloaded just sort of the janitorial aspects of the server to this other chip and then it handled all the network and all that sort of thing. It's called nitro. So this gave them a sustainable cost advantage where Microsoft is offering an instance that runs on intel chips, Amazon is offering an instance that runs on intel chips. But because Amazon can fit like 20% more virtual machines on one chip, that means their cost to serve is structurally lower than Microsoft's was because they have this whole co processor sort of thing. And this is how. This is just. This is a great example of Amazon in a nutshell and why they are the anti Apple. And I mean that in a very positive sense. Apple's all the way on the extreme of yeah, we're going to highly differentiate our products. Like we're going to keep our OS is going to be exclusive to our hardware. We're going to have a developer so we get network effects ecosystem. We're going to have brand, we're going to have all those things that make Apple Apple. That gives us a structurally sustainable. And we're going to differentiation.
- [16:50]
A
They'll just charge price for 20 years.
- [16:53]
B
That's right.
- [16:53]
A
All the competition.
- [16:54]
B
Amazon's all the way on the opposite side. They are going to invest a ton and ton of money over years to build structural cost advantages. So in commodity markets. So they can do things which they
- [17:08]
A
did in retail too. Yeah, they retail to do the same playbook.
- [17:11]
B
That's why they're watching satellites. Right.
- [17:13]
A
So like one question though, as far as that history is concerned, as Amazon optimized for cost structure and serving it more efficiently than some of its competition, did they then charge lower rates and take market share that way? Is that how AWS took over the world? Or was it something else.
- [17:34]
B
Well, if they were just first in general, number one. Number two, they've had, they've always had, like way more features. Everyone else, just because they keep building features and it's an 8020 thing where everyone complains about us and all this stuff's hard to use and they're like, oh, they should cut everything else, but keep this one thing that I want. Thing everyone needs is great. Everybody else, right, yeah. And you. And everyone goes out and they're like, I am not going to get locked into a cloud. I'm just going to use commodity hardware, a basic compute instance, a basic storage instance. So I could take it from Amazon and go to Azure if I want to, or go to gcp. And then you're like, you're developing. You're like, oh, I could spend a few months building this, or I could just use this API that's helpfully there from Amazon. That will solve this one problem. We're just doing this one thing. Don't do it too often. We don't want to get locked in. And then you fast forward and you're totally locked in. You're all using all their services. You're not going anywhere. This is actually a super important point. But if you're not going anywhere, Amazon has pricing power over you. They have features that price power over you. And if there's not enough demand or if there's not enough supply in the market, they're going to have, you know, a lot of pricing power. But this is actually an important point. The big shock when AWS was revealed and I called it the AWS IPO like a decade ago was everyone sort of, they kind of got the cloud but assumed it was going to be super low margin. And it turned out that actually, no, the margins were great, especially for it from an Amazon perspective. You know, I think at the time, I want to say it was something like 17 to 19%. People thought it'd be 1 to 2. Now it's in like the 30s or something like that, or maybe, maybe even higher.
- [19:10]
A
So turned out much higher margins than the retail business.
- [19:13]
B
Right. It turns out you could get not. So Amazon. Yes, I'm talking about this very compelling total commodity market. People could switch wherever. The reality is everything's a mix and there are moats and you do lock people in and you do. And so. But yes, you can also offer stuff super cheaply if need be in a competitive, like in a bank offer. Like, they can go. They go to companies. That's why they give. They go startups like hundreds of thousands of dollars in credits. Like you're not even paying anything to Amazon unless you're actually a substantial business. They will make long term deals with companies. Okay. Sign up for three years will give you like an 80% discount like just because you're locked in. They could do that because they have this, this capacity to do so.
- [19:53]
A
That's one of the great advantages. Okay, so I, I'm going to repeat my question. Has Trainium risen from the dead? Okay, so story look better?
- [20:02]
B
Does this a very. We're, we're doing a wandering here and we're going to make, make it back. Yeah. So the, so the concern that, that that semi analysis article raised. They'll put a link in the show notes to this article. It was, it was really interesting at the time is Amazon is so committed to and locked into their proprietary networking in particular that they can get like Nvidia stuff and they can plug it in like I think it's HGX racks or whatever and so you can access an Nvidia instance. But actually the future is these huge. It's not just a chip and it's not just a rack or eight the HDX I think were eight GPUs eight G. It's entire racks and it's not just entire racks, it's entire data centers that are linked together. This is what Jensen Huang goes on and on about about Nvidia is like it's like the entire data center as a GPU and it's this huge sort of thing. And that, that is totally Amazon's whole strategy doesn't work. You have to do all Nvidia's networking you have to do. And so the concern that was raised that article is Amazon is so committed to their strategy particularly in terms of networking that they're going to miss fall behind more and more on AI as networking becomes more and more important and this operating at systems aspect becomes more and more important. And it's not that that article was wrong.
- [21:29]
A
Okay.
- [21:30]
B
That is all true in terms of training. Training needs this horizontal scaling, this super low latency between chips, between systems and Amazon has never been a big player in terms of training. It's been Microsoft, it's been Oracle, it's been Elon Musk and Xai building their own data center that is dedicated to doing this. And what Amazon got right is training for a long time dominated the amount of compute that was used. Even post chatgpt for several years like more like 60% of chips in the world are being used for training. Not for inference, but when and if inference came along, the needs would be different. In an inference world, you're mostly keeping everything within one chip. It's like you want to get. You want to. This is where the model distillation stuff comes in. You want to get everything on one chip and you don't even worry about too much of the horizontal networking. And it's all about, like, batch size and getting stuff in. Like all these people are coming in. And this is where the CPU aspect gets more important because you have. There's more orchestration. Like this job goes there, just goes there. You're trying to just keep these GPUs filled. What you're not doing is running these huge horizontal clusters that go that, that go across, like just. It's just a different structure, it's a different data center setup to be serving inference than it is to be doing training. Sure.
- [22:59]
A
And it makes intuitive sense that as far as inference is concerned, that would be a more commodit space where efficiency matters more than performance. Where.
- [23:09]
B
Right. Well, that's where you're actually selling, right? That's right. Yeah. You're actually selling. And in theory, if all this training is going to be worth it, it has to shrink as a proportion of compute because the training manifests in a model that is used for inference.
- [23:26]
A
All right, and that is the end of the free preview. If you'd like to hear more from Ben and I, there are links to subscribe in the show Notes or you can also go to SharpTech FM. Either option will get you access to a personalized feed that has all the shows we do every week, plus lots more great content from sirtechery and the strikeri bundle. Check it out and if you've got feedback, please email us at emailarptech fm.