← Big Technology Podcast
Big Technology Podcast
How Amazon Rebuilt Alexa From The Ground Up — With Panos Panay and Daniel Rausch
00:54:42
Loading summary
Transcript175 lines
- [00:00]
Alex
The Amazon leaders who spearheaded the new Alexa are here in studio to talk about what it took to rebuild the pioneering AI and where voice AI is headed in the age of large language models. That's coming up right after this. Welcome to Big Technology Podcast, a show for cool headed, nuanced conversation of the tech world and beyond. We are joined today by Panos Panay, Amazon Senior Vice President of Devices and Services, and Daniel Rauch, Amazon's Vice President of Alexa, for a fascinating conversation about what it took to rebuild Alexa effectively from the ground up. Gentlemen, so great to see you. Welcome to the show.
- [00:39]
Panos Panay
Thanks, man.
- [00:40]
Daniel Rauch
So great to be here.
- [00:40]
Panos Panay
It sounded kind of fun.
- [00:41]
Alex
You both must be relieved to have this out.
- [00:44]
Panos Panay
Yeah, I mean, excited. Relieved is a tricky word on this one. You know, we're finishing the product now, it's coming out next month, so we're, we're pumped that we're through the event. And yeah, there's some relief, I would say. Would you agree? You feel a little bit of relief, but the truth is like, it's all about getting it in a customer's hands as fast as possible. So you still, the team's feeling that urgency right now.
- [01:07]
Daniel Rauch
Yeah, that's the big moment for the team. Right. You get that first customer response. So we still feel like we're building towards it, but yesterday was great.
- [01:14]
Alex
Okay, so I have three Echo devices in my house. We have three rooms.
- [01:17]
Panos Panay
Yeah. What are they?
- [01:18]
Alex
House is generous, but in my apartment, there's one in the bedroom, there's one in the kitchen, dining room, and there's one in the office. Yeah, first generation. I'm really looking forward to getting these updates working hopefully within these devices and getting a chance to use a new and improved Alexa. I've been hanging on to the Echoes for a long time in the hope that something like this would happen. So we're here and I was at your event where you were announcing it. I'll give listeners a little bit understanding of what I saw and then we're going to go into some questions about what it was like to build this. So this new Alexa, it's called Alexa, it is conversational, so it understands natural language, it understands your context, and you don't have to say Alexa every time. It will sort of have a back and forth with you. It is, I think you could call it agentic. It allows you to take action, like book a table, call an Uber. It will go out on the world and help monitor ticket prices for you, for instance. And it's also deeply integrated into Amazon Services and Namely Prime. It's going to be free for prime members. $19.99 a month. If you're not prime members. Not a Prime member. And the coolest thing I saw in the demo was that I think one of you asked for the song with, what was it? Bradley Cooper and Lady Gaga.
- [02:32]
Panos Panay
I didn't say Lady Gaga, just said Bradley Cooper.
- [02:34]
Alex
Bradley Cooper in that. What was the movie called?
- [02:39]
Panos Panay
Star is Born.
- [02:39]
Daniel Rauch
Star is Born.
- [02:40]
Alex
Star is Born. Great movie, Great movie. And then it called up play the song and then you said, now let me see it in the movie. And it connects to prime video and you could see it in the movie. So very cool product. Definitely. I think what a lot of us have been hoping to see from the Alexa team and from Amazon on Alexa, we're going to talk a little bit about what it took to build it and then the strategy here. So I think the first question I need to ask you both is what did take so long? Because I think that like, for all of us who've, you know, I think there's 500 or 600 million Alexa enabled devices out there. We've been wondering, as OpenAI's of the world and other companies have made these big advances on voice AI when Amazon was going to make its move, and you have made the move, but what was the process that made it take as long as it has Panos?
- [03:31]
Panos Panay
I think the easiest way to say it is when you have hundreds of millions of customers that are active right now. We talked a little bit about it yesterday, but every one of them matter. How do we make sure they all get the great experience they need? Meaning you can't start from zero and ignore it. And if you could, it could be much faster. Although it's not that easy to hook up the thousands of APIs and all the partners that we're bringing together and all the experts, it takes time. But the first thing is there's two parts to it. But the first thing is you got hundreds of millions of customers, they love certain things that they do on Alexa today. They might not love everything, but they love certain things for sure. You can't leave that behind, can't wake up one day. And whatever you use Alexa for, whether it's timers or music, you can't not make it better and great. And so you don't feel like something was taken away from you. When you take something away from a customer, you've just missed, you've missed. And so that's one. It takes time to make sure you can get it all done. So everything on what you would call Alexa, not Alexa, works on Alexa, but better. And that was just the first point part of the vision. Can't leave anyone behind, which was important. We can talk about devices and so forth, but customers who love their products that are in and they need them, we can't take that away. That was one second piece is you're re architecting from the ground up, so you've got first the weight of keeping hundreds of millions of customers, and then you're re architecting from the ground up. If we started from zero customers, I think this is a different story. You can move a lot faster. We can solve problems and then just add features as we go, if that makes sense. So maybe we just had a conversationalist, a pretty cool one. Then we can add personalization, then we can add memory, then we can add the experts. And people would just get updates along the way and maybe learn and be great. However, on day one, we need to support everything people love and know about Alexa day one. And so a little bit of patience there and takes a little bit longer. And the vision was, the vision was clear. Like we're going to go bring a conversational agent forward, an assistant for everyone that is smart, has memory, can personalize to you, and then ultimately be incredibly useful. And so when we had that laid out, we, okay, great, but we can't leave any customers behind. And right at that point, you kind of step back. Once you put the vision together, you realize you need a full RE architecture, but you're not going to leave your customers out. So you're rearchitecting pretty much two stacks at that point. One, what is classically known as Alexa, to be awesome and come into this conversational world. And the other is everything new that it has to do.
- [06:16]
Alex
Yeah. And I want to go a level deeper with Daniel on this one because Panos, what you're talking about, a RE architecture is sort of what I've heard has been the holdup here with Alexa for all these years, which is that. And Daniel, tell me if I'm wrong, but basically what folks have told me is that the old version or the original version of Alexa was built with a lot of like, if then commands. Right. So you know, it will understand some structured commands, turn on the lights. Okay. Then it will take that and almost like deterministically say, okay, I understand this command. This is what I'm going to do. Turn the switch. With large language models, it's a completely different ballgame because you have to make room for uncertainty. So actually the fact that you've been able to introduce an Alexa with large language models, which I think will be able to keep that functionality as an engineering feat, that's my perspective from the outside. What is it actually like on the inside and how close is that assessment to the challenge?
- [07:12]
Daniel Rauch
Well, the team will love to hear you say engineering feat because I do.
- [07:15]
Panos Panay
Think that, I think there's no lack of feet. It is real.
- [07:19]
Daniel Rauch
That is the size of the task for sure. I think, I think you're onto it for sure. You know, large language models. The one thing I'd add just in terms of thinking through the technical architecture to what Pano said is that it's really just the latest generations of large language models that can even do the things that Alexa needs to be able to do. So you're talking about our Nova models, right? Which we announced within the last few months and starting to get into customers hands. It's super exciting, you know, partnership that we have with Anthropic. You really need very state of the art technology at the base of the architecture in those large language models. And in large part because of what you said, we need them to behave in ways that we can predict and are certain. Someone says lock my door or you know, play that song, you want it to happen, right? Some are higher consequence than others and you really need to get it right. But you also want all the elegance and nuance and understanding and non deterministic behaviors of large language models themselves, right? So we would call that a stochastic system that, you know, it's literally at runtime that you're making those determinations. So if you want to integrate tens of thousands of services on day one, day one out of the box, take advantage of everything that Alex has always been able to do, as Panos was saying, and introduce all of this new unbelievable behavior that you can get out of large language models. That is a big engineering feat.
- [08:44]
Alex
So how does it know when the user is saying turn the lights on versus like something more esoteric, like is there something built within the technology that's kind of like a switcher that determines first your intent and then decides which part of the model to send it out to?
- [09:00]
Daniel Rauch
The way to think about it is, you know, at the base level you have large language models and you have this model agnostic system that's even itself going to choose the right model for the job and the models play different roles in there. What's already happened is even honestly sort of in the way you asked a few of the questions is that people Assume the large language model is the product. A product like Alexa is so much more than just a large language model. So you have models playing many different roles in the system overall, even models helping us decide which model and models themselves, deciding if they're the best model, you know, tool for the job, so to speak. So then you have a system that progressively decides how to get something done. I wouldn't think about it like a switch or something. In classic computer science, that is a, you know, it's a gate. That's not, that's not how the system works. It's a collection of model behaviors and systems downstream of that that complete specific tasks. And that's where we introduce this term expert to try to help coalesce around the system behavior and explain it better. The large language models are interacting with these experts that do things like get you the sports score, play a song, play a video, know where you are in the song so that you can go to the video, like all the things that you saw yesterday at the event.
- [10:18]
Alex
And so panos, this is a mixture of experts model.
- [10:20]
Panos Panay
It is, you think about it, and a mixture of experts model. But each expert theoretically has its own model as well. So you're building on top of it. Each expert is smarter. When you think experts, it's a weird term. Yeah, but there's think photos, smart home entertainment, whether that's music or video, local info, info, all the partners that connect. You have communication expert, you have an artifact expert, you have a memory expert, you have a personalization expert. Each of them play a role and they kind of arbitrate with each other at all times.
- [10:52]
Alex
So like, the model is just lighting up when it determines that's what you want to do.
- [10:55]
Panos Panay
That's right. Daniel kind of said it well, because the LLM at the bottom of that stack is deterministic. It's choosing which model to use. Then the experts come into play on top of it. It's pretty. It's a pretty phenomenal way to, you know, it's a pretty interesting way to think about it.
- [11:10]
Alex
This is a mixture of experts model for those at home. It's been part of what Deep SEQ has used to become much more efficient in its reasoning. For instance, because instead of lighting up the entire large language model, it's deciding to light up certain areas that might be. I mean, it's not a Deep SEQ innovation, but they've just kind of used it to an extreme extent. Has using that architecture helped you build this in a way that's, for instance, like reducing latency or sort of lightening the compute burden that you otherwise might have had.
- [11:41]
Daniel Rauch
If you want something incredibly fast, stable, even secure, like the paths on data. Right. Where you're really taking care of customers, this is the fundamental approach. I think that that is state of.
- [11:55]
Panos Panay
The art and accurate.
- [11:57]
Daniel Rauch
And for sure, don't forget accurate.
- [11:59]
Panos Panay
So important.
- [11:59]
Daniel Rauch
Yeah.
- [12:00]
Alex
But on that note, I mean, are the, is the new Alexa, is there going to be some sacrifice to having those Alexa commands, those standard turn the lights on, set the alarm in order to enable all the LLMs to work the way that they. They're going to?
- [12:16]
Panos Panay
I think you just called out the sacrifice and it's time. Okay, how long it's taken us to. To get to where we are? That's why it's my favorite question, like, why is it taking you so long? If I told you where we were four months ago on somebody said, lock that door, and then we had to determine what that meant versus in the past, lock my front door. And you had to know it was the front door and you had to say, front door. It's pretty phenomenal. But you know, six months ago it took longer than anyone would wait to lock a door. And you know, our customers need immediate response and we won't make that trade off. So to be that accurate with the latency that's needed with the speed sub 2 seconds, at the end of the day, you end up needing a little bit more time refining the expert so the expert can be quicker and the model can pick the right model quicker and the smaller model can be trained to make sure it knows where the door is. He gave an example earlier, which I thought it's a nuance, but let me just share it with you. Previously in Alexa, you couldn't say, play that song. It would look for a song called that.
- [13:26]
Alex
Right.
- [13:27]
Panos Panay
It was that simple. Now the model has to reason and say that song. I wonder what he's asking. I wonder what she's asking. I wonder what the person's asking. That's what's happening in the system. Then the expert shows up, looks at the history, the personalization. What conversation were we having? Play that song. Oh, he's talking about the conversation we just had about Bradley Cooper and Lady Gaga. Shallow, Shallow play. That all happens in, you know, sub 2000. You know, how many milliseconds are we talking?
- [13:58]
Daniel Rauch
We count in single milliseconds now in system component.
- [14:01]
Panos Panay
So now all that is going on and the stack's working through it versus today, which is play shallow. And that's the only way you're going to play shallow. Yep, that's it. And so I think it's just understanding that nuance in where natural language comes in, where you can talk to the, you can talk to Alexa without being precise, just like you can talk to me. And I'll use some microtels to get, you know, are you asking me a rude question, a great question, a nice question? Are you leading me? And then from those microtels, I can then move to the words and then determine where you're taking me. And you don't have to write it down, type it and read it. Exactly. All that is happening now in the machine, which is pretty powerful.
- [14:42]
Alex
There was a cool scene in your demo at the launch event where I think Panos, it was you, where you said, don't play the music in the baby's room.
- [14:51]
Panos Panay
Yeah, so I didn't say that. So that's very explicit too. Right? Don't play the music in the baby's room. The model will come up, the expert will show up, the music expert. This is where it's super powerful and go, got it. Play it everywhere else. Or you can just say, don't wake the baby, play the music everywhere. Then the model will go, don't play it in the baby's room. That's cool. I know what they're asking. So this is where just that small model in the Expert does its job. And the fact that you can just naturally move it around in that demo, I don't know if you noticed, by the way. Nerve wracking.
- [15:25]
Alex
Yeah. So for listeners, Panos did this entire demo live. I mean, we're going to talk about Apple Intelligence in a second, but Apple Intelligence. I was at the WWDC launch event and it was all a vision. And what we saw at this Alexa launch event was a working demo. Now look, I mean, we know to reserve us commentators know to reserve complete judgment until it's in our hands.
- [15:46]
Panos Panay
Yeah, you have to. You have to for sure. But it was real. It was all real and working. Yeah, but what makes you nervous in an event like that, you're not worried about the product working. I mean, six months ago I would have worried about the product working and I would have shown you more vision demos, like videos, but the product's working. The challenge is the infrastructure. The thousands of WI FI signals that are pinging around that room. Like it's just an unusual. These live environments are very unusual. Turns out tech reporters like tech and they're using a lot of it.
- [16:21]
Alex
We're all on the WI fi More, More.
- [16:23]
Panos Panay
I mean, the signals that are being pulled from Bluetooth to WI Fi to. I mean, who knows what's in pockets.
- [16:29]
Alex
And one of my favorite tech demo moments is Steve Jobs just losing his shit on stage because all the reporters are connected to WI Fi. And he's like, you could either be connected to WI Fi or you can have a demo. You pick.
- [16:41]
Daniel Rauch
Totally.
- [16:41]
Panos Panay
I mean, I'm not. You know, I think bloggers have a right to blog, but if we want to see the demos, we're not going to be able to do it unless we turn off all these MiFi base stations and laptops, set them on the floor. Yes. That's not.
- [16:59]
Alex
We didn't have to have that situation.
- [17:00]
Panos Panay
And then you got, you know, the servers have to be lit up and you're worried about latency and what's happening in the room. So you got all that going on. And now you're going to do live. And this is your baby, right? I mean, you love what you're about to show. You love it. And if it doesn't go off, like, I don't want to tell you what the backup plan was, you know, what was the backup plan? We're not going to talk about it for real.
- [17:21]
Daniel Rauch
Let's not talk about the backup plan.
- [17:23]
Alex
Let me just. Either way, you can't tease that and then not share the backup plan.
- [17:27]
Panos Panay
They were really good.
- [17:28]
Daniel Rauch
They were really good. It was not a great backup plan.
- [17:31]
Panos Panay
No, it was. They were great. They were great. They weren't going to work, but they were great plans. I would say I'm looking over here at some of the team that was helping yesterday. I. But during that moment, you may have heard, it was very nuanced. At one point I said, move the music. Bring the music here. I want to hear the music over there. And the reason I use different sentences, I know what the model's going to reason over and do, but I wanted to make it clear you don't have to think about what you want to happen. You just have to talk. I want the music over there. Okay. And if the model doesn't know, or if Alexa doesn't know, she'll ask you. Do you mean in the living room?
- [18:19]
Alex
Yeah. So are we going to have a speed trade off here from the traditional Alexa tasks? Just quickly, Daniel, I'm curious. Is the stuff I was doing beforehand like I'm doing now? Set an alarm. Is it going to take a little longer because of this process or it'll be the same amount of time?
- [18:33]
Daniel Rauch
No, I mean, this is where we have such a high bar before we're willing to put it out because. And deterministic systems are incredibly fast.
- [18:43]
Alex
Right.
- [18:44]
Daniel Rauch
It is straightforward computer science in this day and age with an AWS cloud and the great connectivity that everyone has in their homes to make a deterministic system fast on something. Exactly like you said, making a non deterministic system fast that can respond in any way gathers all the context, figures out legions of different things, which experts to invoke. Making that system fast on something as simple as an instruction or, you know, is hard. That is quite hard.
- [19:11]
Alex
What technological breakthroughs or innovations did you rely on to get it from a place where you were dissatisfied with the latency to a point now where you're happy?
- [19:21]
Daniel Rauch
Well, I think it's another version of using the right tool for the job and building a system of that's frankly just more complex overall to get the simple things done. So it's a bit, you know, there's like an irony in that. But you need a system that creates very fast paths for simple things. Even though you started with an incredibly complex system, already you're adding these kinds of complexity to get simple things done. So that, I mean, I won't go into the specific technical details here, but that's the upshot. You need to be able to figure out you're trying to do something simple so that you can do it fast. It's a very complex and it gets tricky.
- [19:57]
Panos Panay
You know, people understand how to speak to Alexa today. I think our new customers, we want to, you know, and current customers, we want to open their minds on what they can ask for and how to, how to get something done. Take the simple tasks that we have, timers, alarms, there's a different way to think about them. And then in the non deterministic world, how to translate what's being said into what's being asked for, which is different. An example you said, how quick will be setting an alarm? It'll be lightning fast and you'll likely set it the way you always have. I need an alarm. Set an alarm for 8am I think that's the classic way to set an alarm. Or you can say, Alexa, I need to wake up tomorrow at 8. Okay, and now that's non determinist. And now it's going, I think you need an alarm and then it'll offer you an alarm or just set it. Same with the timer. Set me a timer. By the way, how long do you want the timer for? You say the time you can move that to set me a timer. To I'm cooking my steak medium rare. And then she'll say, I'm setting you a timer for six minutes.
- [21:05]
Alex
Okay?
- [21:05]
Panos Panay
And so it's. You understand, like when you get into that natural language, non deterministic, what's happening? What are you asking for? You're cooking your steak. Okay, I'll get you six minutes on each side. Or tell me how thick it is, and then the answer is, you know, 2 inch thick, whatever, or I want a ramen egg. That's eight minutes. I got you. Tell me when you start. I'm starting. Eight minute timer started for you. And so the world just changed from even these most simple tasks. It just changes. In the spirit of, by the way, I never knew how long it took to cook a ramen egg. So I'd always have to go to TikTok, open it, spend 20 seconds watching somebody make ramen eggs, and then eventually it says, put it in the water for eight minutes. That's all you see until the next week. And then I would say, yeah, it's very true, by the way. Don't search ramen eggs. You get hammered with ramen eggs. But I think. And then all of a sudden you're like, got it. Eight minutes. Set a timer for eight minutes. Now just change it. Just ask for ramen egg. And Alexa will just determine what you're looking for and give you an eight minute timer.
- [22:05]
Alex
Okay? So just to wrap this section on the technical side, my note that I wrote to myself that said they spent too much time building the Alexa microwave and the Alexa alarm clock and not focusing on the technology. Maybe I underestimated the technological lift here a little bit.
- [22:22]
Panos Panay
I don't know. We can't determine what you were thinking for sure, but I think there's a lift here. You said it's a feat of engineering. That's where you started. We have one of the best teams on the planet working on this. A lot of it has 10 years of history in it. There's so many people that work on Alexa today that have been there since its inception. You've got a lot of passion around that in the engineering team and the product. You know, just the product team all up. We call product makers, when you put them all in a collection. And yeah, it's a feat. It's okay, though. It doesn't matter if somebody thinks it should be easier. It's not easy or whatever. It doesn't matter actually, if it feels like it's easy. That sounds pretty good to me, right? I mean, I don't mind. It Means the customer's happy. Like, this must have been easy. Like, yeah, okay, I don't care. Do you like it? Like, do you love it? Great. I think that's where we go.
- [23:11]
Alex
So I want to talk about the vision of this product and the strategy that you're going to put into play here. Because again, I was sitting in the audience and I talked about Apple Intelligence before. I guess this segment of our conversation is. I have headlined it's Apple Intelligence, but it works. And it's a little facetious, but I.
- [23:31]
Panos Panay
Tried not to read anything you posted coming in today because I was like, oh, no, I don't want to defend or have a preconceived notion. So that's interesting. Do you have to keep sharing?
- [23:39]
Alex
We've been talking on the show a lot about how. We talked a lot about the buildup to wwc, the reveal, and it seems like every big tech company has almost the same vision. And tell me if I'm wrong here, but Apple was like, the Apple Intelligence demo was like, you talk to Siri and ask when your flight is and you're switching flights and it's helping you pick your kids up. And that demo looked a lot like the Google Assistant demo that I've seen, like almost every year at Google I o. And then I saw your demo and I was also just like, this is a similar idea, which is that it's a contextually aware, smart AI assistant that helps you get things done and makes your life easier. So I'm curious if you both see the competitive landscape in the same way. I do, if there's something different about Alexa than the others and how you plan to win, given the landscape is.
- [24:44]
Panos Panay
Developing the way it is, you want to jump in. I got a long one here, so why don't you just. No, you start and then I'll go.
- [24:53]
Daniel Rauch
I mean, look, the vision for Alexa has been super consistent actually for 10 years. I think Panos, this was it made it into your final deck, I believe, yesterday. You know, we have always wanted to just make lives easier and better, simpler, and be the world's best personal assistant. That's been the vision for Alexa from the beginning. And so now we just have a technical leap that lets us get closer to that vision. But nothing, you know, that's been the vision since. For all 10 years that Alexa's been out there. We have a much more capable AI assistant that's conversational, that is personal and personalized now that can get an incredible amount of things done for you. But the vision's consistent Okay, I want.
- [25:39]
Alex
To go to Panos in a second, but I need to follow up on that because the reaction to this reveal has been, this is great. It's personalized. It has your data to help you figure things out. But then you look at a company like Apple, which has so much personal data that people have trusted Apple with because it has this security messaging, or Google, which, you know, has your, you know, maybe your Gmail, your Google Calendar, Google Maps. This is. These are the services that you use to get around the world and interact with people. So if you're going to be this personalized assistant, like you are coming up against these companies that basically have already been deeply integrated into people's daily routines. So what is the play there?
- [26:22]
Daniel Rauch
I mean, the phone. You're basically asking about the role of the phone.
- [26:29]
Alex
Not just the phone, because Google has plenty of services on the desktop. I mean, I'm on an Apple machine. I got Gmail open Maps to figure out how to get here Calendar. And so it's almost the operating system for your life.
- [26:44]
Daniel Rauch
I mean, look, you told us you have echoes in every room in your home, and that's great.
- [26:48]
Alex
That's also true. I'm starting to think there's too much tech going on.
- [26:51]
Panos Panay
Well, you might look at your job.
- [26:53]
Daniel Rauch
I mean, come on, if you didn't, this is.
- [26:55]
Panos Panay
This would be a problem.
- [26:56]
Daniel Rauch
I'm just saying customers, you know, we do so much for customers in the home today, and of course, we're Amazon, so that's not just. Thank you, by the way, for having echoes in every room in your home. That's awesome. But also, we probably put some packages on your doorstep and probably stream you some content, and we've got great, deep relationships with our customers. Prime is an incredibly valuable program, for example, and, you know, hundreds of millions of customers literally take value in that and love it and use it all the time. So we love our relationship with our customers, too and think that we can deeply integrate any services customers want as well. We work with Gmail, we have the Outlook Calendar. We integrate Apple Calendar.
- [27:37]
Panos Panay
I think it's a very powerful point. You have to take that and understand we're both kind of a. We have this, if you will. You have music, shopping, movies. This is real things that people love doing in the home. I mean, these are personal at every level. Photos. But also. But we're such an open platform with thousands of partners, it's hard to say it's a platform, so I'd be careful with the word. But at the end of the day, every Single integration point across Alexa gives us so many of those insights as well. But the key. And Daniel hit it when he asked you a question. It might have been rhetorical at some level. I don't think there's anyone close to being able to understand your home as Amazon, as Alexa. It's a super important element for us, Alex, the idea that smart home is connected to your music, to your entertainment, to your life. The fact that we're now bringing in memory to Alexa and you can have that conversation, it'll hold the context for you. I don't think there's anything else like it, because then it's connected to all your services in a natural way, too. I don't think it replaces the centerpiece of the phone. I think it just adds value to your life in a very different way. And I think there might be a little bit of opportunity. And this is me understating it, but the ambient devices in your house right now and the ones that you can buy from us and some of the beautiful products that we're both making now and have released recently, they're in your home. And you don't have to think. You don't have to open anything. You don't have to log into anything. You just have to be there and speak. And it's a powerful concept when natural language shows up.
- [29:28]
Alex
Yeah. I was speaking with Jamil Gandhi, the head of prime, at your event yesterday, and he was talking about how the family calendar is on his Alexa device and it is a Google Calendar. So the fact that there is that interoperability, I think, where you don't have a phone, that actually might. Maybe that's an advantage. I'm just trying to.
- [29:48]
Panos Panay
It is an advantage. Just think of it this way. Like, we're not asking you to start something that you knew that you don't already do.
- [29:56]
Alex
Right.
- [29:57]
Panos Panay
We just want to make it simpler for you. So Google Calendar is a great example. Okay. Just attach all four of your family's calendar. We'll make it a family calendar and put it front and center for you. And then when you decide if you're going to dinner on Friday night, we'll rationalize it. And you know that concept that there's a communal device in your house that everyone can see, you know, it's something that people have been asking for for a long time, but now that you have so much intelligence in the product and it can do the rationalization for you, I feel like we stand alone there.
- [30:29]
Daniel Rauch
I do think. I think this calendar example is one that helps flip the Question a little bit in my mind because it really is like, how often do you say, well, it was just on my calendar, I didn't know to meet you there. Why I was on my work calendar. I say that to my wife Tali, you know, all the time. She's like, we missed the restaurant, we missed the reservation. So anyway, having one spot that can be communal and personal, pretty powerful.
- [31:00]
Alex
I want to press a little bit on this because the phone seems to be the place where people, it's all about where do people interact with these assistants. The phone seems like it's going to be a pretty important place.
- [31:12]
Panos Panay
It will be.
- [31:13]
Alex
So if you don't have a phone, I mean, again, there's some advantage in that you can bring any service in. But if people are on an Android and they're summoning a Google assistant, whatever the name is that week, or they're on an iPhone and they're summoning Apple Intelligence or Siri, where does Alexa fit in on that? Are you going to have to look at deeper integrations with these phone makers? Will they even allow you to do that?
- [31:39]
Panos Panay
I think people use different assistants. I don't think there's any question about it. I don't think there's one. Although if you lean into Alexa, we have the Alexa app on the phone and with one touch of the button on your iPhone, you're having the same conversation. You're actually carrying the conversation from your home to your phone to your car to your PC. With Alexa.com we thought that through because we needed that thread for sure. So as she becomes more personal to you and then more needed, you want to have her with you everywhere. That app is doing a crazy cool job right now. Now we haven't released the new Alexa app yet. It's coming with. If you get Alexa, you get the Alexa app. The Alexa app as well as alexa.com Alexa.com right.
- [32:18]
Alex
There's going to be a web version of this.
- [32:20]
Panos Panay
There is. And you just do the more traditional long form work that you do with any AI browser at this point. It's the easiest way to say it, but you also get all the personalization. You also get the context of carryover. If you had a conversation in your kitchen, it'll just remind you what conversations you've had lately. If you've booked a reservation, whatever you've done, it'll collect it there. So it'll be on your PC and your phone as well. So I think we just want to provide that for our customers so they have the opportunity to say I want my assistant, my single assistant with me everywhere. You might use your phone for different things. You might use a different AI assistant on your phone. I think that's a fair proxy. I don't. I wouldn't disagree. It just depends on what's the best path to get something done. I think Alexa will provide a lot of that best path.
- [32:59]
Alex
Okay, I want to take a quick break and then talk a little bit about the agentic elements in your new Alexa release, where agents might be going. And then maybe we dream a little bit about where this technology is going to lead. We'll be back right after this. We're back here on big technology podcast with two Amazon executives responsible for the new Alexa. We have Panos Panay here. He's Amazon's senior vice president of devices and service, and Daniel Rauch is Amazon's vice president of Alexa and Fire tv. It's interesting that this agentic buzzword is now starting to be translated into things that we're seeing in product. And it's kind of interesting because Alexa has had skills for a while like call me an Uber, and now you can use Alexa to call you an Uber. So is this actually like a really a new moment for agentic AI, or is this rebranding of some stuff that works a little better than it has Panos? What do you think?
- [34:02]
Panos Panay
I can't get it to work anywhere else. I mean, I think this is. At the end of the day, it's incredibly new, but it's also solving so many different things at the same time. First, you have to always go back to how much understanding is in an utterance, just in natural language, being able to translate it. And we've talked about this already, getting down to calling a service, calling the right API, making the right partnership so that API is called to make it as simple as possible. I don't think it's been accomplished. I don't think you're seeing it out there anywhere connected to an assistant right now. I think there's a lot of maybe you've seen it, you got to share with me where it is, but I don't think you have. I have not. And so what agents fundamentally like using a core LLM with an agent, non deterministic. Calling the right API, calling that service, booking that service, bringing it back and tying it back into all your other services.
- [34:59]
Alex
It's a demo we've all seen a thousand times, but haven't been able to use, I think, as consumers.
- [35:05]
Panos Panay
Yeah, okay. I think, yeah, maybe that's the case. I haven't seen those demos myself, but I do, I believe it. I believe it. I maybe need to watch Closer, but I do think it's new. I think it's new what we've created and what we're doing and building it up. I think it is.
- [35:22]
Daniel Rauch
I also think we might mean different things by agent. And so I'm just curious, Alex, just to make sure we're grounded in your definition.
- [35:31]
Panos Panay
For sure, there's a grounding difference between us.
- [35:33]
Daniel Rauch
Just in passing, I mentioned yesterday in my own part of a part of our event, you know that, boy, everyone just uses this term agent. And I do think people use it in different ways. What does it mean to you?
- [35:42]
Alex
Yeah, it's such a great question because I do think that in some ways that agent has been used to rebrand automation. We've been seeing automation demos forever. I mean, even so, just to give you one example, I wasn't trying to shade the Amazon demo. I was just to give you one example. Yeah, we were all. I mean, we. A lot of folks watching the tech world were at Google I O when they demoed a voice assistant that will go. They'll call a restaurant for you and book you a table. And like, they did the actual conversation. And the. The assistant has like human utterance, goes, well, maybe we could have a table for. It's like. And then it was to go.
- [36:18]
Panos Panay
And that's interesting to the restaurant.
- [36:20]
Daniel Rauch
I don't, I don't. I just don't remember using it, but correct.
- [36:22]
Alex
So again, there's the demo. There's the demo and then there's real life. But I think it was also just like you gave a tech command and it would go out and do that for you. But a lot of this stuff, like I said, we've seen demos. We haven't seen it actually work. My definition for agent is something that can go out and accomplish for you. So you had a good demo that I enjoyed watching about trying to go see a Red Sox Yankee game. By the way, for folks listening, the reveal event was in New York. Daniel's apparently a Red Sox fan. He trolled the entire audience, including the guy sitting directly.
- [37:00]
Panos Panay
There was a guy wearing a Yankee suitcase. It was almost like he planned it. I kept saying, are you sure you want to do this Red Sox bit? He's like, for sure.
- [37:09]
Alex
The entire off season acquisitions, which Alexis, I mean, as a Mets fan, I will say you were fine.
- [37:14]
Panos Panay
You were fine, by the way. You saw that. You saw the info expert in action right there.
- [37:18]
Alex
That's what it was.
- [37:19]
Panos Panay
Yeah. Because you're now in. It was not deterministic. And then, of course, it's a different answer every time, Alec. Every time Daniel did the demo. At the end of the day, I mean, it was Alexa's decision to talk about Alex Bregman. It wasn't Daniels. Like, you couldn't lead that. You can't plan that. And so a bit of a risky demo, because if Alexa decided not to talk about Bregman, I don't know where you would have taken them.
- [37:42]
Daniel Rauch
I do know a lot about the Red Sox, so I figured, you know, maybe eventually we get to buy some tickets, is what I was thinking. But it was to set an example of that kind of agentic capability and sort of set the baseline of what we mean, which is, hey, I actually was just having a chat about the Red Sox. Could I get some tickets? Actually, that's a tough game to get. Oh, they're expensive. Can you watch for tickets for me? I mean, that was where we ended up with the demo. Could have ended up in a lot of different places. But being able to set an agent off, if you want to call it an agent, in that case, we think about it a little bit differently. But in that case, that agent capability to say, first of all, I could buy you these tickets right now. Second of all, you don't like the price, I'll watch for you. Infinite Patience never runs out of gas. If those tickets do drop below a certain price, I'm notified and can buy them. That's a hugely useful thing for a customer.
- [38:36]
Alex
Yeah. And you could buy it with a command.
- [38:38]
Daniel Rauch
Yeah.
- [38:38]
Alex
Because you're integrated with Ticketmaster.
- [38:40]
Daniel Rauch
Exactly.
- [38:41]
Alex
Yep. So, yeah, to me, I would say that's agentic behavior.
- [38:46]
Panos Panay
Great.
- [38:46]
Alex
I would say it qualifies. We had some questions. We have a big technology discord. I was sharing notes with the crew as the event was going on, and we had some notes from people about what they want, sort of beyond those simple use cases. I mean, I call it simple, but obviously there's a tech lift to get it done. So one of our listeners said, is Alexa still going to be reactive to requests, or can it be proactive and suggest at the start of the day some smart ideas based on the context that Amazon has? For instance, I would say, you know, do I need to order any birthday gifts? And it would then go out. Go out and say, well, look on your calendar, there are, you know, five birthdays coming up. These are the dates, and these are our suggestions. So is it going to. Is it. Because that's, I think, A step further.
- [39:40]
Panos Panay
I think you're stepping in. You're stepping. You said you want to talk a little bit about the future and how proactive Alexa can be. Like, there's a balance. One, we think Alexa can be incredibly proactive. Like, to the point of when you wake up in the morning, you walk into the kitchen, it's like, Alex, you didn't sleep well. You know? And then you can imagine integration with some partners that is like, okay, let's have the conversation. Also say, hey, your day looks pretty packed today. You should probably find some time that proactivity is there. It's in the system. We're using it in a very different way. We don't want to be intrusive with it. We got to learn from our customers first. Like, how much proactivity do you want? I think it's very, very important to, you know, you don't want to jump to that future. You got to be right. So, yeah, it's a good example. Wake up in the morning, and if I need to buy a birthday gift, can you just remind me? We can create reminders. We can create a conversational piece, but I don't think a lot of people want Alexa just to wake up and start talking to you.
- [40:32]
Alex
No, I do think that. Yeah.
- [40:34]
Panos Panay
Don't want to be intrusive. You got to be really careful. We got to be so smart about, you know, we have 10 years of lessons. This is what's so awesome about it. And how much privacy matters. And when you want to invoke Alexa to be part of the conversation versus how proactive you want it to be. And we have a balance on it, But I think it's a good push. She's already proactive in the spirit of. She has a way to. If I went out there and said, hey, I've been looking for this. I watched this movie last week. What was that song that was playing in that movie? Okay, give it that little information. Check prime video. What was he watching? Okay, I got it. I think you're watching this movie. It was this song. Proactivity also includes. Do you want me to play that song? Or you just want the name of it? And a lot of times Alexa will say, do you want me to play it for you? That's a subtle proactive. It's not intrusive. It's using context. Contextual information, some memory, some of your history. And in the past, you've asked me to play it every time. So why don't I just ask you to play it? I think those are different forms of proactivity. But our vision includes Alexa being proactive. It has to be that we believe the next step customers will ask for is, I want her more, not less.
- [41:46]
Alex
Right.
- [41:47]
Panos Panay
And so instead of me thinking, oh, I should ask Alexa, is there a point where Alexa will know to ask me? I think that's a real question. I don't think that's today. I think that is the future. And I think back to where we're pretty well positioned for that. If that's what customers want, I think we can do it for them.
- [42:05]
Alex
But I think what this listener was asking is, can I just, with natural language, say, you know, take a look at my calendar and tell me something?
- [42:13]
Panos Panay
Okay, so that's different. That's different. Sorry. I went all the way to my vision, but here's what I'll pitch back. That already happens.
- [42:19]
Alex
Okay.
- [42:20]
Panos Panay
So when you wake up in the morning, whoever that listener is, here's the answer.
- [42:22]
Alex
Yes, with Alexa plus or with Google Alexa.
- [42:25]
Panos Panay
Okay, sorry, not with Alexa.
- [42:26]
Alex
Right. So this is.
- [42:27]
Panos Panay
There's no way it's gonna happen with Alexa. Okay, it's not. But with Alexa, 100%. Wake up in the morning, get your daily brief, tell me what's going on. And you know, Alexa knows what time you start work, will warn you of the traffic. You should probably leave by 8:20 if you gotta be there by 9 today. Like, that level of proactivity that's in the system, but you have to engage first.
- [42:47]
Alex
Okay. This idea of Alexa being proactive like it is, it's definitely. I see where your caution is coming from, because there are these proactive notifications that you get with Alexa. I've had to turn them off.
- [43:00]
Panos Panay
Yeah, yeah, we learn from that.
- [43:02]
Alex
Yeah. So, okay, that's good that there's learning there. I could go with some other Alexa product feedback, but I feel. Let's use our time.
- [43:11]
Panos Panay
Let's stick with Alexa for a minute. But if you want to talk about Alexa and we can tell you if Alexa has fixed your frustration, well, the.
- [43:17]
Alex
One thing I'll say is I use it to play alarms. And there have been moments where it will play the ad before it will play the song in the. But that kind of goes to a question that we did also get in the discord where people talked about. They talked about, whose assistant do you trust? And in the back of some people's head, there will be this perspective. Amazon is just going to try to sell me something. Like, for instance, that example of, you didn't sleep very well. All right. It's like A suggestion for sleeping pills coming up. I don't know exactly what it is, but how do you get past this perception of I'm going to get. Because you do with an assistant, you trust it with a lot of data. So how do you get to the point where people are comfortable sharing this data and feeling good about the fact that it won't be used to lead to purchases?
- [44:11]
Daniel Rauch
Well, I mean, first, I think even before you get to that part of the question, it's just, how do you manage a customer's data? How do they see transparently what you're doing, what they've told the system? Do they have control over their data? So all of that's so paramount that you have to start there. Actually, it's like one question earlier than that, which is, do you trust Alexa? And the answer has to be yes. So we've been building on a foundation of transparency and control. There's the Alexa Privacy Dashboard, which one great place to see everything in terms of system settings and your data, etc. I just want to make clear all of that carries forward to Alexa. I think that's sort of the important point to make at the top. And then if the question is, is the question, boy, should I be, you know, should I be offered a product in a given case where a system thinks I need it? I find that great when it's great. It is great when it's great. Like, I found a pair of shoes. I don't even think it was on Amazon recently through something I was reading online and I've been, I've got orthotics and you know, it's great when it's great. Basically, I was referred something. They're awesome ultras. They have a wide toe box. I'm not gonna sell ultras on your show. I'm just telling you that I found them because if you're listening, we need sponsors.
- [45:33]
Panos Panay
Is this the camera? Which camera?
- [45:35]
Alex
That's the Altra's sponsor. Yeah, pilots give them a heads up.
- [45:39]
Panos Panay
We aren't sponsors. Alex needs sponsors.
- [45:42]
Daniel Rauch
It's an arcane example, but the bottom line is like, it's great when it's great. And why is it great? It's contextual, it's relevant, it's offering me something that I actually need. And so building systems where you can do that elegantly, customers actually love that we get feedback that that's great. What's terrible is when you get inundated with things that are irrelevant to you. And so we're building a system that doesn't do that.
- [46:07]
Alex
Does Alexa need to have a screen? I mean, Panason put this to you. You're the head of devices at Amazon. A lot of the demos that you did at your launch event were with Alexa with a screen. Again, I have like first or second generation echoes in my house. Might be time to upgrade.
- [46:21]
Panos Panay
You should upgrade. Like, there's a couple of things you're missing. One, you're missing speed that you could have that you don't have. And I think speed is time. For me, it's comfort, you know, it's confidence. Like there's so much like first. Yeah, I would always encourage. Not just because I want to sell the next device. That's not why. Just having something modern. If your device is nine years old, you're missing eight years of tech, okay? So I'm judging you, giving what you do, you know, and so your feedback is like half heard at this point. But I would say, I say that, you know, jokingly, but I go, look, you need a more you. It's better. The product's just better as it, you know, generationally, generation over generation always got better.
- [47:03]
Alex
Now does it need a screen?
- [47:04]
Panos Panay
Incredible. Yeah, it does.
- [47:05]
Alex
It does. Okay.
- [47:05]
Panos Panay
It doesn't have to have a screen. It's a better experience with a screen.
- [47:09]
Daniel Rauch
Okay.
- [47:10]
Panos Panay
It really is. Now let me qualify it because you have a screen in your pocket that works with Alexa. You have a screen on your desktop that works with Alexa. The screen in your home, you should have one. It's very powerful, it's nuanced, it's not intrusive. The new design is elegant, it's soft, if that makes sense. Like it's what you want in the home, something softer. You can get the expression from Alexa, from that screen, and she brings visual expressions as much as anything. But here's the trick. It will come with you in your earbuds. It'll come on your Alexa frames. It'll be in your pocket, it will be in your car. So you don't always need a screen, but in your home. I mean, the command and control, the information management, what you get off of it, it is powerful. Will it work without a screen? Absolutely, absolutely. And it'll be great. So need is a relative term. I want you to have a screen, okay? Because the experience is that much better and there's a nuance in it. Like when we start rolling out preview, the first customers to get preview will be our screen based customers, because it's the best experience, that simple. And so you'll be like, I want the preview and I'll say, get a screen. Get a screen.
- [48:29]
Alex
All right, maybe two screens, and then.
- [48:30]
Panos Panay
I'll light up all your. We'll light up all your echoes. But you need a screen.
- [48:34]
Alex
Okay, maybe one in the kitchen, one in the office.
- [48:36]
Panos Panay
You only need one. You only need one. Yeah. I mean, it's up to you, but.
- [48:39]
Alex
Screen out the bedroom. At least that's my perspective.
- [48:42]
Panos Panay
Totally.
- [48:42]
Alex
Like, the only screen I allow in the bedroom is the Kindle.
- [48:46]
Panos Panay
It's a cool product, but I'm using mine here, so. Good listening to you. By the way, I got the alarm in the morning Note. I get that bug file I got you. But the idea that different devices work in different places is real. But I think you need a central hub right now. I think Alexa is so dynamic, and the more you can learn to do, the screen will teach you. Like, hey, get after it. You saw Daniel's thumbtack demo, which is a little bit even. Was more agentic than, if you will, for us, than the GrubHub. Did we do GrubHub or OpenTable last night? OpenTable.
- [49:23]
Daniel Rauch
OpenTable, yeah.
- [49:24]
Panos Panay
With Uber.
- [49:25]
Daniel Rauch
Right.
- [49:25]
Panos Panay
But the thumbtack demo was, you know, conversation. Let's. I need a repair person. Well, that agent goes out and starts booking it for you on the website. And then you need the screen to give you a status, like, working on it. Back in a bit. Don't worry about it. I think that's what you want. That ambience for in the background. So I think. Can't be more clear. I don't think. I think it'd be great.
- [49:47]
Alex
Okay, I'm sold. I'm going to get one. All right, we're running up on time here. I want to give you both a minute to answer this question, and then we'll head out. But it's got to be a minute or your team here will have my head. We talked about how voice AI might be the future of AI or the catalyst for these large language models on the show. A while back, OpenAI, for instance, debuted or introduced this advanced form of AI called with GPT4O. And you can see the inflection point of ChatGPT that the second they announced that, bam, it goes from 100 million to 300 million users. Is Voice AI the future of artificial intelligence?
- [50:25]
Panos Panay
I'll start and I'll close this out.
- [50:26]
Daniel Rauch
I mean, we've believed for a long time that voice is the most natural interface. We're using it right now. We're using it with your listeners, we're using it with each other. It's incredibly expressive. You can load an unbelievable amount of context and power in it. You can be definite, you can be vague, you can be nuanced. And it's just we're born with the knowledge of how to use it, and it's completely intuitive. So I think we do strongly believe that it's one of the best ways to get things done. It is not the only way to get things done, but I do think it's pushing us, it's challenging us to get more and more human, more natural, and that's why it's always been one of the kind of centerpieces of our vision for Alexa. So, yes, my answer is yes, and I think it's really pushing the envelope now.
- [51:11]
Alex
Okay, a minute to you, Panos.
- [51:12]
Panos Panay
I think we're at that time. This is the inflection point and mentioned it yesterday. You know that I believe the vision for Alexa is incredibly ambitious. It centers around voice, for sure. I don't think it ends at voice. I think the interaction model needs to be the one that's most natural to you. No doubt. If you need to touch the screen to complete a task, if you need to get to your computer and write the long form, I think it's a flow. And the thing you don't want to do is you don't want to block the customer from the interaction that they need to go get something done. It's why we're on the phone. It's why we're on the PC. It's why we're in your glasses. It's why we're in your ears. And ultimately, though, the anchoring point of all of it is the voice, because it is natural. It's innate to all of us. The trick is getting to natural conversation. The trick is trusting that you can just talk and realize that as we talk to each other, it's pretty sure you can talk that way with Alexa, and you're going to find that. And I think that is the transformation that's coming. I think it finishes the next chapter, ends the first chapter and starts the next chapter and leads us to getting. So finishing is the wrong word there, but getting us to that next leap over the next 10 years. This is that starting point. The technology is enabling it right now, and that inflection's happening, and it's compelling. So it was a longer way to say, yeah, it starts with voice, but I don't think it ends with voice. It never will. Like we. It is also innate to us. You always. We as. As humans, we're always going to find the best and easiest path to get something done. And we think voice will lead to most of that, but not all of it. Like, we don't overstate it. Like, we will find the best, easiest, which means basically the fastest path to completion, which is why you need to upgrade your devices and get a screen. Are you with me?
- [53:03]
Alex
I told you already, I'm buying one.
- [53:04]
Panos Panay
All right, we'll get on it, man.
- [53:05]
Daniel Rauch
We did sell. We sold at least one device here in New York. Good news.
- [53:08]
Alex
Thank you.
- [53:08]
Panos Panay
While we're here, our goal this week was not to sell devices, but it will do that soon.
- [53:12]
Alex
This is very efficient and scalable.
- [53:14]
Panos Panay
We're killing it. Now we have a new sponsor. We sold the device like this is. We're killing it.
- [53:20]
Alex
Well, look, Panos and Daniel, I want to just say while we're recording that I don't take it from for granted to be speaking on record with Amazon. It's always great for me to be able to hear what you're doing and be able to ask these questions, and I'm sure for listeners, it'll be great as well. So thank you both for being here.
- [53:36]
Panos Panay
And thanks for coming.
- [53:37]
Daniel Rauch
Thank you so much.
- [53:37]
Panos Panay
It's been really great.
- [53:38]
Alex
Awesome. Well, thank you, everyone, for listening, and we'll see you next time on big technology Podcast.