← Just Now Possible
Just Now Possible
Building AI Employees for Hospitality: How AITropos Takes Orders Where Customers Already Are
01:07:41
Loading summary
Transcript142 lines
- [00:04]
A
Welcome to Just Now Possible with Teresa Torres.
- [00:09]
B
Hi, my name is Santi Marciori. I'm the CEO at iTropos and I'm an engineer MBA. I've been doing product software product for the past 10 years. And I'm an AI fanatic. Absolute fanatic.
- [00:24]
A
Love it. Juan.
- [00:27]
C
I'm Juan Aedo.
- [00:29]
D
I'm CTO at Itropos.
- [00:32]
C
I'm a software developer. I've been doing software development since the age of around seven or eight. I'm 42 now. I got my first computer when I was a little kid and never stopped getting into it. And with that in mind, I've always been working with cutting edge technology, always looking for the latest things. Of course, AI excited me as soon as it showed up. I have a degree in data science and yeah, I have a lot of experience, more than 15 years of experience in hospitality software for hotels and restaurants. Been working on that. Yeah, that's how we got where we are now.
- [01:08]
A
Amazing. One of my questions I want to dig into is why hospitality and how you found this space. But before we get there, tell me a little bit about what does Atropos do?
- [01:19]
B
Yes. So we are actually building AI employees for the hospitality industry, and we're trying to generate real operational impact. Right. So it's not just a bot, not just a chatbot, but it has a lot of tools and we have many integrations so that we can get done real operational work. That's what we're doing. We're addressing restaurants and hotels mainly, but we also have bakeries as customers, we're looking into bars. And every hospitality business can use our AI employees, basically.
- [02:00]
A
Yeah, I love that. Okay, so we've had actually quite a few companies on the podcast where they're basically creating AI employees in one way or another, whether that's like customer service agents or. One of our recent episodes was with a company that's like creating agents to help with managing clinical trials. I think this is a very hot space right now because AI is capable of so much, but it's also a little bit of a tricky space. Right. There's a lot of fear around our jobs going away. And I think, especially in the hospitality industry, I could see there being. These are the types of businesses where it's probably hard to find good employees, especially if we're talking about bakeries and restaurants. The owners are probably tapped out and need help. How do you think about this balance of what's good for AI, what's to do, what's good for humans to do? I think especially in hospitality, I think of hospitality and I immediately think service. Like when I show up to a hotel, I like having a human welcome me. Do you want to just tackle some of this? Like how do you think of these hard challenges?
- [03:08]
B
Yeah, and that's an amazing point. And of course, the AI employees are not for, for every type of hospitality business, but there are four. There are super useful for a lot of services. For example, we are one of our niches is qsr, quick service restaurants. Right. Because in those cases you don't go for the human attention, you just go for food.
- [03:34]
A
Yeah.
- [03:35]
B
And the same thing applies for hotels. There's some hotels that of course you have to have the human attention, the human touch. And there's other hotels that you just want to get some room service or that you just want to schedule the taxi for to the airport. So that's where we believe those are our targets. They are super distinguished. And the customer, the restaurants and the business owners really understand when these type of services are useful for customers and when they might need the human touch.
- [04:10]
A
Yeah, if I'm sitting by the pool and I need a margarita, I'm okay with ordering from an AI.
- [04:14]
C
To your point of where it impacts on places where this technology can be used, the places we're looking to do it is not replacing where a human should be, but where the technology is in the middle, in between humans. So basically, instead of having a human to human conversations through a platform, we know that we can replace that interaction with human to agent or human to AI through that same platform.
- [04:42]
D
Right.
- [04:43]
C
So even more so, some other places where we think we can be very competitive is where technology even doesn't have a person on the other side. So for example, when you're using an app, we are basically focusing on the conversational interface rather than the app interface.
- [05:02]
D
Yeah.
- [05:03]
B
And maybe just to finish wrapping up the idea, our product, of course we market our product as AI employees. We're even thinking about changing that because AI is not our product. Our product is delivering a high quality service for guests and restaurant customers. And we actually do that, of course with a lot of AI. But sometimes we need humans to jump in. Right. So our main focus is to deliver an amazing experience. And one of our goals is to pass the trim test. The idea is to make customers feel like they are speaking or interacting with a human. And we are achieving that in many cases, in many cases, people, the final consumers thank us and even send us pictures, food pictures to thank us. And they, they think that they explicit think about how the attention was super close.
- [06:02]
A
Like this framing. It's interesting. Let me back up. I can see clearly, like in hotels, there's a lot of rules where I could see your service being really helpful. You already mentioned room service. I joked about the margarita by the pool, but anybody who's been to a resort that's busy has had this experience of you just can't even find a human. Your example of a walk up restaurant makes a ton of sense to me. I'm curious about the restaurant category in particular. You mentioned the bakery. These are more experiences where I feel like we're used to talking to a human. And I know like here in the US During COVID a lot of our restaurants moved to QR code menus to reduce human contact. But we're seeing, at least where I live, we're seeing all of that go away. Like people want to connect with humans. But I also know restaurant owners that can't find good employees and they are struck. They're like working two full time jobs and still struggling to run the business. Tell me a little bit about where do you see this playing a role in restaurants? Just so I can get a clear picture of what types of rules your product is filling.
- [07:11]
B
Yeah, so let's jump right into a specific example. Yeah, let's say McDonald's today. You have to either order through the kiosk or wait in line to be served by a human. Imagine if you were able to just get into the McDonald's, take a seat and order through a voice message on your phone. And of course the employee will tell you when your food is done so that you can pick it up at the counter. Or if the restaurant can have runners, they will take you the food to your place, to your, to the table where you're seated. That's one of the main use cases that we're aiming for.
- [07:55]
A
Yeah, I can see this being really powerful. Like, what immediately came to mind is my town has a big concert venue and outdoor amphitheater and I would love to be able to just order a beer and have it come to me rather than walk into the tent and standing in line. Yeah, amazing. Okay.
- [08:13]
D
Line.
- [08:14]
B
There is a huge potential for AI employees to taking care of their orders.
- [08:20]
A
Yeah, that's a great way to think about it is why do we stand in line?
- [08:23]
B
Correct.
- [08:25]
A
All right, Juan, I want to go back to something you said in your intro. You said you've been in the hospitality industry for 15 years. Is this how the two of you landed in this space? Tell me a little bit about how you found this as the area you wanted to work in.
- [08:41]
C
Yeah, actually we met with Santi on a previous company we were working on that was mostly related to marketing, but did marketing for hospitality basically for hotels. But yeah, I got into that company because of my experience also working with hospitality systems. So basically I worked with the company here in Argentina for around 15 years. Actually it's more I always say 15 because I'm used to using telling that but it's five years ago was 15 so we could say that it was.
- [09:12]
D
And I'm still sometimes working with them
- [09:14]
C
like I do consulting for them so that, that kind of still there. But this company basically builds one of the PMS softwares property management software system which is what hotels use for, for their operations. It's one of the, one of the systems that has the biggest market share in Argentina. And they also have of course a POS system which is for restaurants. So I've been working with those systems for a lot of time for all of these 15 years that I mentioned. And that got me into kind of understanding how the business or that market works. So after meeting with anti and working on all of this, of course it was mostly it went on its own decanting on that specific market. Because of my experience, Santi, I know he can tell but he's also been working a lot on that on the industry as well. Previously he's been working on companies. So we both not had, we both had the knowledge maybe from different areas, but totally related. Both super nerdy about the cutting edge technology, super nerdy about AI. And so it was a no brainer to get started with this. Of course there were just because of the contacts we had or what we've been working on for these a lot of years we had a lot of easier entrance into the market than if we had thought about, I don't know, going into, I don't know, automotive, automobile market, which I know nothing about cars.
- [10:47]
A
Yeah. So it sounds like you both had a lot of domain expertise in this area. You were geeking out on the technology and just excited to play in this space. One thing that's interesting to me is you have a very broad problem space. So hotels have a lot of employees, they have a lot of use cases. Restaurants have a lot of employees, a lot of use cases. Tell me a little bit about how did you decide what to do first?
- [11:18]
B
That's definitely a great question. And we did decide to work in the hospitality industry, but actually before that with Juan, we spent two years meeting with industry experts and analyzing hundreds of different Ideas. And we finally met someone with a lot of domain expertise in the restaurants area, in the restaurants industry. And that's when we realized that there is one specific use case, one specific feature that can unlock an immense potential, which is order taking. There's lots of companies, lots of people helping with chatbots, providing information, etc, etc, but there's this use, this specific use case which is order taking orders, which is super hard and super valuable. So actually we do have different employees, but our main feature is being able to take orders. And that's how we actually decided about it. We actually started doing an assistant for waiters. That was the first thing that we started building. We literally spent six months working on a device, on a physical device that waiters would use. And while working on that, we realized that taking orders was the hardest part. So we said, okay, let's focus on this. And then when we started offering this solution to different restaurants, to different potential customers, they started asking for this service to be deployed in for customers directly. And that's how we found out that there was huge potential for a solution like this for customers. So we pivoted and we started focusing specifically on that.
- [13:12]
A
And were you using AI already at that time?
- [13:18]
B
I can't remember a time when we didn't use AI anymore. I don't even want to think about that. Gives me the chills.
- [13:28]
A
Yeah. I think if you're familiar with the product market fit question of how disappointed would you be if this product went away? I feel like AI for people that have embraced it, like disappointed is the wrong word. How devastating would it be if this technology went away?
- [13:43]
B
I think I wouldn't be able to breathe.
- [13:46]
A
Yeah, I know sometimes I wake up and anthropic has downtime and I'm like, how do I do my job?
- [13:51]
B
Today we had a few episodes.
- [13:55]
C
That happens to us. Well, we, when we run out of credits, for example, we use AI a lot to code. As you might imagine, that helps us move faster. And as soon as we run out of credits, it's low.
- [14:06]
D
What do I do now?
- [14:07]
C
Do I have to code manually? No, no.
- [14:10]
A
Yeah, that's usually when I eat a meal, I'm like, just step away from the computer. Go have a meal, go outside.
- [14:18]
B
Yeah. And it's fun because we literally are the first ones to find out when some of the LLMs is not working. We jump X and there's nothing there. Like silence. Five minutes later, a thousand tweets.
- [14:31]
A
Yeah, okay. So you know what I really like about your story is you clearly had domain expertise. You still took a lot of time to figure out the right problems to solve. You found an area that was a little bit that looked promising. You started to build in that space. Your customers, it sounds like, pulled you even further. And forget waiters do this for our customers. Santi, you said you spent two years looking for problems to solve. Was that full time? Were you both working somewhere else? Tell me a little bit about that exploration space.
- [15:06]
B
Yeah. And two years is an understatement, to be quite honest. I spent the last 20 years thinking about startup ideas. Right. But the two years was specifically related to Juanu and myself, both of us together. When as soon as we met, we started enjoying very much our conversations and started thinking about so many ideas that we could build. Actually, just a very quick. Both of us love astronomy and at some point we fantasized with building a company that is called tas, which was Telescopes as a service.
- [15:42]
A
Nice.
- [15:42]
B
We were trying. Our idea was to use SpaceX to put in orbit a telescope and lease the time of the telescope. So that's how we literally spent a lot of time together thinking about ideas. We were working full time, so yeah, we did it on our spare time.
- [16:01]
A
Yeah. I hope you someday also make that telescope company. I think that would be fun.
- [16:07]
B
Yeah, absolutely.
- [16:09]
A
Okay, so let's get into this a little bit. You got pulled into. You were first making software for the waiter to make it easier to take orders. This got pushed into, can we just give it to the customer? The first thing I love is that this is a very specific use case. You're not looking at a hotel and saying, let's do all their jobs for them. You're saying let's take orders. The other thing I love about this is, Santi, you mentioned this was a hard problem. And I can imagine you're not just spinning up a knowledge base and being an answer bot. You've got to integrate with point of service system, point of sale systems. I'm imagining you're interfacing somehow with a kitchen that maybe is making food or something real in the physical world where that order is turning into something real. So give me a sense of what does it take to solve a problem like this? What's the big picture?
- [17:02]
B
Yeah, so the connections, the integrations, all of that, although they are quite complicated, those are not the hardest parts. The hardest part is being able to translate the not deterministic world of human conversations and LLMs into a structured information so that you can feed that to systems. That is one of the hardest parts. That's what took us A long time. You can do a prototype in a day. Anybody that uses AI can do a prototype in a day. But making sure that this is consistent and that works every time takes a lot of time. That's one of the things that we realized when we started working. And how did we solve that? Putting a lot of hours. Putting a lot of hours. Understanding and making an architecture that is super advanced. I'm gonna let Juanu speak more about all the architecture that these agents are using. They are not a simple prototype that you can a day.
- [18:13]
A
Let me make sure I understand where you said the challenge was. So there's first you mentioned the non deterministic human, which I love this because we all talk about the non deterministic LLM, but it turns out humans are also non deterministic. And there's this like first layer of. I'm assuming this is a chat interface. I can say anything.
- [18:33]
B
Yeah, yeah. You connect it to WhatsApp.
- [18:36]
A
Okay, yes. So that's the first challenge of the user can enter anything.
- [18:41]
B
Yeah, yeah.
- [18:42]
C
The platform is actually channel agnostic. So basically we started with WhatsApp. Here in Latin America is like the OS, the operative system of Latin America. That's what we use. But we could easily connect to iMessage, SMS, any other channels and it's part of our roadmap as well and it's the easiest thing to connect for us. We're just using WhatsApp right now because of that. But one thing I wanted to mention regarding what Santa said about the challenges was all of the things that Shanti said, not only they had to be correctly done, but as you might imagine, since you're doing real time order taking, the agent has to be fast and responsive and respond correctly while in a timed fashion where it doesn't get the customer waiting. As you might imagine, agents as agents today are mostly used for long running tasks.
- [19:36]
D
Right.
- [19:37]
C
And that takes a lot of time to process information. So one of the biggest challenges, and I had something like here on my ear constantly. We need to look. I remember, I don't know if you saw the Playlist series based on the spot, how Spotify was built. Okay, so the Playlist is a series where it shows how Spotify was built from the different perspective of the builders. And one of the things you could see is how the person had the idea, I can remember his name constantly had the technical guy saying faster, I need songs to load faster.
- [20:09]
D
No, the time between a song and
- [20:10]
C
another song has to be faster and faster. That's how I felt with Santi on my side. But that paid off because we're actually right now at a place where, like Santi said, people are not noticing they're chatting with an agent, not only because of the response, the way the agent responds, but also because it responds not too fast, but not too slow either. And that's kind of part of the challenges that we were trying to solve.
- [20:36]
A
I love that you mentioned not too fast as part of the challenge, because I know, like, when I send a support email and I get a really detailed response, one second later, I'm like, yeah, that was an AI. Okay. So it seems like there's this first layer of the human gets to enter it whatever they want. So you gotta deal with the messy of the messiness of this. You have an agent that is trying to understand that message. I imagine your agent is doing the heavy lifting of interacting, integrating, like, structuring that input in a way that works with your now deterministic systems. Point of sale. Whatever. So give me a sense of. I want to go back to, like, day one. What was your first prototype? How did this start? How did you even evaluate if AI could do any of this?
- [21:25]
C
Yeah, that's something you can start. But one thing I want to say is that the core piece was always the same one, which was basically have an integration with this external system. That was something that remained along these different iterations. But I let Santi talk about the first product we built that we iterated. Actually, we had two. I think we're on the third iteration right now.
- [21:50]
D
Right.
- [21:50]
C
So until we have the hardware, then we have the custom app for waiters, and now we're actually at the.
- [21:58]
B
Yeah, yeah. I can't remember how many iterations we've done on this product, to be honest. But yes. So we always. We were super optimistic about this, Teresa. We were super optimistic. We saw the potential of AI and we actually never thought that this couldn't be done. I think it was our. Our determination to make this work, which made us push very hard. And again, the first time you. One of the best things about AI is that it gives you some very quick dopamine hits. Because making a prototype, it's awesome.
- [22:40]
D
It's awesome.
- [22:42]
B
But that is good and bad at the same time because. Because you have no freaking idea what you are, what you're starting to do, what you are getting into. You have no idea before you start, but it gives you this dopamine hit, and it's like you feel a superhuman and you're convinced that you can do it. That was where we Were. But I honestly had some doubts along the progress the process. We spent a few months working on it and we still had an unacceptable error rate because we wanted to make this perfect. And that's when we started. Yeah, testing different ideas, playing around with so many tools, so many different agentic architectures. We have five different types of reg rag with different. It's. It got complicated. At some point we started thinking about the physics and how it should evolve, et cetera, et cetera. But yeah, it was super, super hard to be able to, really to understand every time what the customer is ordering, especially in different restaurants that probably have products that are quite similar. So if you fit that to a prototype, that's when you say, okay, this might be harder than it looks. But yeah, but it was a hard process. But right now it's working so good that we are very proud of what we've built.
- [24:17]
C
Yeah, one. One thing to mention is, and this is anecdotal, but like Santi said, sometimes you could be like overwhelmed at things not working and like you start having questions. Like Santi said, we never thought that this couldn't be done. We just were thinking whether we were at the right time. And I have this memory, I have this snapshot of a chat I was having with Santi, chatting with him, where we were doing this second iteration.
- [24:46]
D
So the first iteration was a hardware.
- [24:48]
C
That was the idea was that kind of like having a headset for waiters where they would just talk to an agent, the agent would help them, et cetera. That was super hard, not only because of the model, but also mostly because of the hardware. Second one was something similar, but on a. On an app, where it was mostly like a chat app. And then the waiter would just make the order, take the order from the customer, make the order on the app, and then generate the order and send
- [25:15]
D
it to the pos.
- [25:16]
C
Like I said, all iterations had the core idea of integrating with the system. Now this third one is with the chat. So when we were at the second iteration, we were trying to get our agent or agents to build an order.
- [25:31]
D
Right.
- [25:31]
C
With orders are super complex objects in data terms, because you have the product, the product can have a variation in the recipe. The product can have something called a modifier, which is like large, small. The product can have extra products linked to it. So you could have like a promotion. If you buy two products separately, they have a one price, but if you buy them together, they have another price. And Depending on which PoS, you have, all PoS have different data structure and that's actually a regular problem on POS system. Like, I tend to think that POS systems are still an unsolved problem because they all have different ways. Each restaurant or each venue has a lot of. Each of their own different ways of doing things each. So they all have in the end to ask for specific custom implementations to the. To the software that the company that developed the software. So there's a lot of. As you might imagine, there's a lot of variance and working on implementation, on integration with all of them. For us, it's super hard, but that wasn't the hardest part. So when we were doing this second iteration and we couldn't get the agent to correctly build the system, I was like, something. And I remember just a message for something. Juanu, I'm not sure can we do this? It's not fully working as expected. And then I was like, trust Santi, we're going to make it.
- [26:59]
D
And in the end, it turns out
- [27:01]
C
that it's about not giving up. Because the only limitation is the technology
- [27:05]
D
I remember having, like I was saying is most of the times is basically trusting your product and understanding whether the problem is that can it be done or not? And if you think it can be done, what's keeping you from it? And it's most of the times, if it's the technology that's limiting you is whether do you think the technology will be there at some point or is it like a physical limitation? I think even Elon Musk works with this idea. Whether if it's not allowed by physics, then it can, it can't be done, but if physics allow it that you can do it, we're super far away from that. But I was, I remember having Santi sending me this message saying, Juan, I don't know if we can do this. It's not working as expected. We're taking a lot of time. And I was like, let me see something. Trust we can do it. And just that exact same date, one of these companies that we use for model released a new model, Smarter. And it was just a matter of, let me try with this. And I just switched the model and it started working without even changing anything from our side. Of course, there were a lot of nuances that we then fixed and improved, and we constantly keep doing that, but it was just a matter of waiting for the improvement on the technology, the base technology that we were using to have our product fully up and running. And that's when we said, yes, this can be done. And the fact that we actually got to the point where the model that we needed showed up for us to get moving faster or actually get moving. Made it okay, man. We're on the right track and we are early.
- [28:41]
A
Yeah, this is. I love this because I think. I think it was Andrew Karpathy said it was either Andrew Karpathy or it might have been Boris Cherney from Anthropic said, to build your product for the model that's coming out six months from now. And what's fun about this space is that like you build something and then time passes. And even if you do nothing, your product gets better. Like the brain behind your product gets better. And it's so cool to see what this unlocks. Like, I had this experience myself with a product that I'm building. I was like prototyping in Claude Code and Opus 4. 6 was doing the heavy lifting. And Opus was great. But Opus is very expensive and you cannot put that in a production product. And so I like switched to Sonnet. I started to play with Haiku. It was okay. And I was like, that's okay. Time will pass. We will get Opus level Brain at Haiku prices. Keep going. And I feel like this is a brand new part of building products. Like, I can't really think of anything else where this has been true, where like, you really can start to see the future and build right on the edge of that future. Which is really fun.
- [29:57]
D
Yeah, for sure. The only caveat I would say is that you got to work. Yes, you got to work on the thing that is coming, but you have to make sure that you build it correctly. Because if you only expect your product to get better just because of the model, that's fine. But that's just adding steroids to something that if your product is not as performant or as good, maybe in the end it will fail because maybe not because it's not working, but maybe because of costs. As you just said, like, intelligent smart models are expensive. So if you only wait for super smart models to come out and you just depend on that, it's going to super expensive. You still have to make a lot of engineering and architectural decisions within your product to make sure that your product should and it does the right thing internally to avoid these other problems that might have.
- [30:51]
A
Absolutely. And of course there's the hard challenge of how do you know what the model's going to get good at and what to delay for later versus what to work on for now. Okay, let's get a little bit into. Give me the high level you've mentioned. Agents A few times. Walk me through. I'm a customer, I'm trying to place an order. What happens next?
- [31:16]
D
So basically you just open up. In our case, we're just going to talk with what we currently have. You just go to your WhatsApp, which is basically something you have installed. That's one of the great things about this approach that we're taking, is that no one has to install anything at all. They just have to use whatever they use. And then you basically go to the contact that represents the restaurant and you start, hey, I want to make an order. And the agent starts talking to you and also guides you on the way on what you want. It even does recommendations for you if you have some sort of idea of what you want to eat. It recommends based on your ideas. It tries to also match products that you like with other things that could go well with that. Of course, there's a lot of marketing from the revenue side that they, hey, I want you to offer this. When they ask for this other stuff, all of that kind of things that a restaurant or a venue would actually train an employee to do. Our agent can do it also. The good thing is that the agent doesn't get tired, doesn't get stressed, they're never depressed, etc. All those benefits. But from there, basically the or the ask the user and the agent chats, basically the agent the steps that it takes. It starts building an order with the products that are being requested. Sets like we mentioned, the modifiers, comments, variations on the recipes. There's a lot of internal processing on because we have many features internally while that happens, which are like, when you add a product, make sure that it has stock. When you add the product, make sure that it has stock. But also you need to check whether the order is for today or is programming scheduling for another date. So if you're scheduling for another day, you have to make sure that stock will be available for that date. And so there's a lot of these rules that, like I was mentioning, have to happen at the same time in parallel most of the times to make sure that we can reply in a fashion. Once you do that, you build the order, the customer is happy with their order. Then the agent basically asks whether it's for takeaway or for delivery. And it's for takeaway, they just ask for the name. If it's for delivery, they ask for an address and we check the availability region for delivery. So the agent can also know whether they can deliver to that address or not. And this cannot happen actually in the Middle of the conversation. It's not something switched to when the order is finalized. It's like when you talk to a person, you just, hey, do you guys do deliver to this area? Yes, we do. No, we don't. So maybe even if the order is not set, it can answer that question. Sometimes even our agent is used for asking questions about the venue, not really ordering, which also works. So that's standard use for agents. And so once you have the order and you have the delivery method, whether it's pickup, takeaway or delivery, the agent basically closes the order and generates a payment link which is also sent through WhatsApp. And this payment link, basically the person just clicks on it, goes to whatever payment platform the customer use, they make the payment, and then on WhatsApp they get a message, hey, we received your payment. Your order is being processed. So after that, because the venue start preparing the order, which also triggers a notification to a customer on their messaging saying, hey, your order is being started working. It will be around ready in around 25 to 50 minutes, whatever that time is. And if it's for delivery, you will tell them, hey, your order is ready and it's out for delivery to your place, or your order is ready, you can come and pick it up. And so basically, as you can see, this is what currently delivery apps are doing, but in a way that the person doesn't have to leave their happy, their safe place, right? Which is their messaging app. They don't have to go to external apps. Here in Argentina, we have these two apps that are for order delivery. I know in the US you have DoorDash and even some venues have their own custom app for ordering, which is a mess as you may imagine. You have to install external apps just for a single venue. So what we're trying to do is, like Santi said, is just improve the experience of the person when taking orders by not leaving their comfort zone, so to speak, for on their phone. And that's basically it. We have the. I think the only time they leave the app is when they actually just click the payment link, which takes them to whatever payment platform it is and then that's it. The remaining of the experience is fully integrated into their chat experience. It even tells you when your delivery guy is at your door.
- [35:58]
A
So help me understand this in the context of, let's say I'm ordering like food for takeaway, am I going to the restaurant website to see the menu? But then I'm ordering through WhatsApp. Like I can imagine if I'm at a Hotel. I have a room service menu in my room. I'm looking at that. I'm using WhatsApp to order. But is that the idea? Like, instead of I'm still looking at the restaurant menu online and then using WhatsApp to order?
- [36:25]
D
That depends a lot on the user. The user can either go to the website to see previously what there is for to then go in order, or they can ask the agent to either recommend them, tell them what's in stock, or the agent can even say, so our agent actually can send attachments, so they can send you a PDF with the menu. They can send you pictures of the food. Of course, as long as they are set up on the system so the full experience can happen on the app. Or they could just go to the website and see the menu. That's up to the customer. We don't force anything on that end.
- [37:00]
B
Where the customer gets the number to speak with our agent depends very much on the business. There are some businesses you just mentioned that you can have the QR code on the room in a hotel, or you can have the QR code print it out on a table in McDonald's, for example, or you can get the number from the website. Or even you, if you're a recurring customer, you have it. You already have the connection. So you just have to go to whatever messaging platform you're using and search the name of that restaurant.
- [37:34]
A
I can imagine, too, for, like, delivery people have their go to spots. Like, I know exactly what I want to order from a specific restaurant, so I don't need to look at a menu. And I can see that being a really powerful use case as well.
- [37:47]
B
Correct.
- [37:47]
A
And I love that it's through WhatsApp, because I don't. I definitely don't want to call a restaurant ever. I'm not a millennial, but I feel like I have that millennial trait. I just don't want to call somebody. And I also, I want to see. I want feedback that my order is correct. I hate placing an order on the phone. I don't really trust that they got my order correct. And I really think the world should just operate over text. So I think this is amazing.
- [38:13]
D
It's basically text. Right. We just have a technology that basically converts the audio into text. But the idea is giving the full conversational experience of DMing through WhatsApp you have with your friends. So whether you want to chat or just send an audio message, which is. Right.
- [38:30]
A
Yeah. Okay.
- [38:31]
B
You would clearly be one. A good customer. A customer that uses this tool.
- [38:35]
A
I Would be. I wish all things could be ordered via tech. Okay, let's get under the hood a little bit. It sounds like there's a lot you're orchestrating behind the scenes. Whether it's checking delivery zones, checking stock, making sure you have all the right data for the point of sale system. How does this, what does this look like under the hood?
- [38:55]
D
Good question. I don't know if I want to tell your secrets. No, just kidding. No. Okay. These are different. So basically what's going on under the hood is just to give a quick sample is basically we have. Our system just receives a webhook or a call from whatever API we use to mess it for the messaging channels. In this case, WhatsApp receives a message and from there starts a full pipeline that does a lot of things. The main challenge was so we actually the first thing we had was that the pipeline was straightforward, like one thing after the other, right. Just to make sure that things worked. But upon iterations on that, then we started saying, okay, we need to shrink down the times. I think the biggest challenge was actually figuring out which parts, for example, could be parallelized. You got, do we talk directly with the POS as we build the order or do we build an external system that takes the order, which is much faster? Of course, the first time we just went through the integration part, next iteration was a no brainer. Let's just do everything inside our app and then send to POS or the integration. The next one was how many things can we do in parallel that can be done in parallel that doesn't need a sequential. Sequential processing. Right. So for example, if you're ordering for. If you're searching or ordering for multiple products, the agent can basically search for all those products at the same time and then build a response based on all the results. Right. So instead of searching one by one, you do a lot of multiple searches at the same time. Then the other thing is like database. The database, basically the database infrastructure. How powerful is the database or the database choice that you use so that it actually has quick results and ordering. But all of that is basically architecturing different ways of treating the data and parallelization caching database infrastructure, database engine, of course, for different things. So all of those things have to be considered at the same time, which is I think the hardest part. But in the end what happens is that the agent in our scenario, we're using tools and we decided to use agent tools other than MCP or RAG because it's the fastest, most efficient way for the agent to interact with mcp, it would have to basically go through the mcp, understand what's going on, make a request to an endpoint to which could potentially be fast. But still it's one extra step with tools is basically, okay, call this function and the tools are already on the prompt of the agent. Now we do some rag, some initial rag retrieval, augmented generation for knowledge basis, or if we want to preload information sometimes. So we have these hacks that we've been implementing where for example, our system prompt is built based on the last message and the previous messages. So we got two system prompts. One is like the main system prompt and then we have something called updated system prompt, which is like an immediate system prompt that the agent gets to know more information about what's going on, but from the system role. And that basically builds the prompt based on, okay, so you're looking for this product. So let's quickly build into that small system prompt information about that product before the agent can respond so that it doesn't have to figure out a tool to use and search for it. We just fitted that information right away because we have to figure that out. Right. So all of this, all of the. Like I was saying, it's mostly architecturing engineering, but just coming up with good ideas on how can you resolve these problems. Hey, it's taking too much time on a database call. Okay, can we do this faster, figure it out on our own programmatically, instead of having the agent figure it out and figure out which tool to call.
- [42:56]
A
So yeah, okay, so it sounds like you have. You started with the pipeline, which I can imagine. First of all, a lot of people on this podcast talk about they start with the pipeline, you have confidence it's going to work, you control more of the process, it's a little more deterministic. I can imagine in this use case though, for the customer, it feels like they're on Rails. It's not a like open conversation where anything goes. It's like the agent is guiding them through their order taking. Whereas one benefit I could see you getting from shifting to an agent plus tools architecture is the customer can drive the conversation a little bit more. Is that what you found?
- [43:38]
D
Yes. So we started using tools in general, like at the first moment. So we consider MCP and workflows such as, I don't know, other workflows with the pipeline. But tools was the no brainer for me and to get started because we did the previous analysis and then we saw that it was the fastest. Now I think that's more what you mentioned is more of an emergence, an emergent property of the fact that we decided tools, this was already happening. The agent has access to all these tools. At some point we did think about using State, just giving state to the agent so that it knows, okay, right now it's just receiving the or greeting the user. Now it's building the order. Okay, now the order is built. So the problem with that is that it would happen what you just said you would be having the customer on Rails and not giving them that freedom of speaking. Hey, wait. So actually there's actually something, a workflow that I can mention that gives a good example. So what you can chat with the agent, start building the order. You're done with the order and the agent sends you the payment link. Before you even make that payment, you can ask the agent to make a modification to the order so they can go back, update the order and they will send you a new payment list. You're not on Rails. You're free to talk as you would with a person on a call center, for example, to take your order. So I think that's. We were lucky enough to make the right call at the beginning early on. For tools. Yeah.
- [45:14]
A
Okay, so it sounds like. So your agent. I'm imagining some of the tools that you. I'm not going to guess. Tell me some of the tools that your agent has access to.
- [45:23]
D
It's just built in tools that we built. So basically the tools are for example add products. You have a tool that is basically create an order, add products to the order. This tool with add products to the order basically has all the logic for adding modifiers, comments, variations, etc. You got check product availability, right? So you got search products. Search products. You got search knowledge space. Of course we have a knowledge base that helps a lot about extra information. We got geolocation tools that helps hey, given this address, figure out whether you can use them. We got generate payment link tool which basically takes care and internally on all of these we have like multiple providers. So depending on the venue's configuration, if they have one payment provider or the other, the agent is agnostic to that. It just goes through it and then the tool takes care of that. That's a good thing about tools, right. The same happens for the geolocation. If you. Most of the times we use Google Places API, but you could use other one if you wanted and it's kind of it we don't. We try. This is a constant back and forth we have with Santi. Most of the times there's A lot of things that go into the prompt, but there's also a lot of things that should be systematically happening. Right. So when the agent tries to take an action, the tool should tell it whether that action is successful or not and why so that the agent can understand what's going on. And this is like I was saying the back and forth with Santi because Santi takes a lot of time and working on super amazing prompts that make the agent talk like a real person. But then we have the okay, does the agent do this? That's when the systematic implementation has to come in. So that's part of where we're mixing with something where okay, it's prompting, but the prompting should be related to what the venues configuration is all about. So we do. We have a prompt composer framework that we implemented which inject fragments and depending on the configuration it's just one fragment or the other then. So we have, we do with Santi constant reviews about whether the current prompt for how the agent should reply contains any logic that should actually be implemented on the system side or it's okay to implement it on the more on the human side or how it should reply. So yes, a lot of pieces in there.
- [47:51]
B
Yeah. Many times we do MVPs for different features through the prompt and then we build the features. Once we realize that it's helpful and really needed by customers, we assume the technical debt to test if that's something that the customers really needed.
- [48:11]
A
Yeah, I love this. This is something that the company every has written about. Are you guys familiar with the company every?
- [48:17]
D
No, I'm not.
- [48:19]
A
Okay, they are. They build themselves as the company of the future. So they have eight different products. Each product's built by one person and they're relying on AI and they introduce this idea of like agent first development. And I don't mean coding, agents writing code first development. But like you let the agent do the feature and then when you see if it's working, you figure out, okay, what parts of this should be deterministic, how do we support this in code, how do we optimize it? But you're really relying on the agent as your MVP to figure out, does, do customers even want this? So it sounds like you guys stumbled on a very similar idea, which is very cool.
- [49:01]
D
Yeah. I have a concept that I've been going through back and forth in my head. I'm still trying to grasp into it, but I'm starting to see ourselves as building these products through how there's business driven design or business driven development test Driven design or test driven development? I think we're on the path of becoming a company that, that does conversational driven design because it's all about. So this conversation has to have this outcome and it's not like the system or this test. It's basically the conversation is what triggers everything. So work on that.
- [49:38]
A
Yeah, I like that. I want to go back to just your tool architecture and there's something you said that I think is really innovative that I want to dig into, which is your agent has tools and it sounds like you're both at the prompt level. You're injecting like I'm assuming your agent needs to know for this company, this is what an order looks like. These are the required attributes. I could imagine that could happen at the prompt. It could happen in the tool. Just making sure that the agent has the right stuff. I can see how this could be a very. It's weird saying traditional for a technology that's three years old, but like standard agent turn based. You get a user message, you call a tool, you do your thing, away you go. But Juan, there's something you said that I thought was really clever. You're interjecting before the agent does a tool call, you're looking at the user message and trying to guess. I want to give the agent more data so that it doesn't have to do a tool call, say a little bit more about what, how you're anticipating there and what's happening there.
- [50:42]
D
So I want to give credit where credit is used. So this was an idea that Santi brought. So we just give it the message of the user to the prompt and have it understand stuff that of course the idea was amazing, but it failed if we just injected the prompt. So I had to figure out a way to do that. And so what we're doing is basically we have smaller agents that are super fast, basically reading either just the message of the person or like the last conversations. And for example, we have one super quick agent that what it does is builds product queries for searching products. So basically figures out whether there's a product being asked on the message. And from that it builds, okay, let's search for this product. And so while other stuff is happening, it goes and queries, creates that query, goes on search and pulls that information. Something similar happens with knowledge base. So for example, instead of to avoid having the agent called knowledge base a lot, we grab the message of the person and we just basically do a search on the knowledge base just to see if the one shot, just a single Shot gives us a result of knowledge base and we also inject that. So we try, yeah, we try to build that. Sometimes it's huge and sometimes there's a lot of information that is not relevant at all. Which is the hard part because the agent starts getting mixed up. But we still looking and we have solved a lot of those, but we're still looking for. So to improve that. But that's how we're doing it. Even we always on that session we only inject the state of the order, for example. So it knows a lot of things without even needing to go through the tool calling to refresh its own memory again and again. It's like we build temporary working memory on the spot instead of having it process. Of course we have memory implementations for. For specific some the current episode, the whole session or even the just the whole person where we do know what the person orders and kind of the suggestions or who the person is. But in the end, yeah, that's the strategy we use for now.
- [53:01]
A
This is I think one of the most fun parts of building AI products which is just this like I to me it's bigger than just context engineering. But it's like what goes in the prompt, what data is most relevant at this moment in time. Even what tools should be available at this moment in time. And almost thinking about your agent as a person and like the way that we would train a person is we don't tell them everything all at once. We really look at based on the conversation what's the relevant information for you right now? Like how do we constrain your space so that you're more likely to do a good job. I could dig into this for the next hour, which we probably don't have time for, but it sounds like you do keep a profile about your customer. So there's context there. I imagine there's a lot of context around. Here's what. Here's this restaurant's menu. Here's all their business rules about what goes with what. On top of all your. The goal is to build an order and what is order look like for this company? Just tell me a little bit about. You must have had to learn a lot just to make all those pieces work together. Tell me a little bit just orchestrating all of that.
- [54:11]
D
I think I have to tell you a little bit about myself. So as I mentioned, I've been a developer since I was 7 years old. My degree is on music, not on computer science. And that's because I just learned to program and I learned engineering just because I liked it and I'm a self taught engineer. So what that gave me is not only I didn't only learn how to build products or how to build program systems, how to correctly engineer architecture stuff, but also taught me how to quick learn by just doing right. And so in this scenario I always been like, let's do, let's test if this works and let's validate and then see or come up with an idea. Just quickly see if that's something that can happen. So for example, this idea of having that system prompt, that immediate system prompt that I was mentioning, like I said, we have two system prompts. We have the main system prompt, which is what people mostly use, but then we have this smaller immediate system prompt. And I came up with that idea, but I wasn't really sure whether that was supported by models, so I had to go and search. Hey, would the model allow me to have two system prompts like a main one at the top and then one immediate injected and then on the history of the chat, that system prompt wouldn't show up anymore. It's just always appends itself on the last message. Is that something that's supported by model? So I had to go through and learn about that. But it's not that. All the knowledge I had for how models worked was already there because I'm always like, see? Okay, this new thing came out. How do you use it? I already knew it even before we can use it. One quick example I want to give on that is just for example, I'm not sure if you heard, but just recently the second version of Mercury came out by Inception. Have you heard of Inception, this diffusion large language model? And I've been on that since their first patient. I was like, man, this is really something interesting. I think this is really the path because it has this property of being able to correct itself. Like previous tokens can be corrected because of just how it works. Instead of having to you write a token and that's it, the token is there. So learning all this stuff gives you the advantage of already knowing what's possible and whatnot. So when you have a problem, you already have the knowledge of the different technologies on whether of what like your tool set is basically the tools that you have at disposal, right?
- [56:53]
A
What you're describing is selfishly why I started this podcast.
- [56:57]
D
Amazing, right?
- [56:59]
A
My thinking was if I collect all these stories, I saw a need, I would see lots of teams trying to learn this. But selfishly I was like, if I interview a bunch of teams about how they Build AI products. By the time I'm building my AI products, I'll have heard lots of ways for how people have already solved the same problems.
- [57:16]
D
That's a great way to do it,
- [57:18]
A
which is really fun.
- [57:20]
D
That's great to see what you build.
- [57:22]
A
Yeah, yeah. I mean, yeah, I have several AI products in the market now, which is really fun all around. Discovery coaching. Okay, let's get into something you said earlier, which is you told me about this moment of doubt. This is really hard. Can we do this? It wasn't good enough. Which really raises the question of how are you evaluating if your agent is working? And I think with orders in particular, I could imagine some failure modes that are really catastrophic. So what are you doing to make sure this works?
- [57:56]
B
Well, we like to stick to one KPI, which is how many items did we identify correctly? That is our most important metric to really understand how well our agent is working. Then it can say strange things or it can use not the perfect vocabulary. All of those things can be improved. But the main thing, main KPI, how many of the items did it get corrected?
- [58:25]
A
That's it. Yeah. I love the simplicity of that. And I think from a customer standpoint, that's probably what they primarily care about.
- [58:33]
B
That's the only thing they care about. Yeah.
- [58:36]
A
Do you see like in your conversations, do you get data to measure that? If you mentioned at the end of an order, they get a payment link, they pay for it, the food gets delivered, they get a text saying it's delivered. If something's wrong, are they adding that to the WhatsApp chat or chat, are they saying I got the wrong thing?
- [58:56]
B
I don't think if it ever happened.
- [58:58]
D
Maybe yes or vice. Yeah, I think that the whole picture here is that the agent can actually take, how do you say, like lamos Santi claims. Claims. Yeah. So the agent can take claims from the customer when there's a problem and it even sends an email to the venue so that they know about the problem. If there was a problem, how it was delivered, they even if the user sends a picture, they can send it. The email will come with all the attachments needed. If there's an error in an order, it's more of a post sale service or customer service that, okay, let's see how we can solve this problem. But to the point that Santi was mentioning, what we tried to do is previously validated and Santi built an amazing tool. And this is amazing. He just built this with lovable. Just a quick mention. Santi is the number one user of lovable In Argentina, in South America, Latin America, something like that?
- [59:56]
B
No, in Argentina, okay.
- [59:57]
D
Within the 1% of users. So they reached out to him to say, ask him. But he built this amazing tool which is mostly front end. Of course, there's a lot of things happening, but basically this tool, what it does it just. He came up with a way, I mean, something you want to talk about it?
- [60:14]
B
While testing it, we realized that if we want to move fast, we want to go fast. There are some things that can go wrong, of course, especially with AI. So we said, okay, there's actually two things that we need to do, because our main goal, again, is to take the orders correctly, all of the items correctly. So we can do that in two ways. The first one is safe. That works every time, which is human takeover. If we do audits of the live audits of the conversations. So whenever we find things that are off, we take over and we correct them by hand. And then we automate that. We fix that problem and we automate it. But before we even get any of the agents into the customer's hands, we do thousands of tests. How do we do that? We actually trained an agent that acts as a customer to test the agent. We run literally thousands of those during the night, when they are done. When one conversation is done, there's another agent that analyzes the conversation and checks if the order was correct, if all of the items were correct. And then after X number of runs, we have another agent that analyzes wherever there was an error, right? And we start fixing that. Of course, the first time we did it, we had a huge error rate, huge error rates. But then we started improving each of the things that were happening. And now, honestly, our production products might make a few mistakes, but then we fix it by hand if it ever happens. Once it happened that we didn't catch the error, and we added an address that was not the right one, one for delivery. But since the customer gets a confirmation ticket, he saw that the address was not right and he called the restaurant. So it actually was quite an easy fix. But there's a whole bunch of agents testing the customer agents so that the items are understood perfectly.
- [62:30]
A
I also realized there, like in my error that I example that I gave, I focused on the wrong food getting delivered. That was because of my bias with DoorDash. I feel like my experience with doordash is I often get food from a totally different restaurant that I ordered from. It's just a weird experience. But I realized in your case, you send them a payment link And I'm assuming they get to see their order on that page. So there's this like, human in the loop before the order is even finalized. Is that true? Yes.
- [63:04]
B
But that human is a customer and he assumes that everything is okay. So they are not testing it. They have.
- [63:11]
A
Okay, so you're not relying on that as a feedback, as a human in the loop?
- [63:14]
B
No, no, no, no. The human in the loop is someone at our team that jumps in case there are any errors.
- [63:19]
A
And how do you literally have somebody monitoring all the conversations? Like, how are you detecting errors real time?
- [63:28]
B
For when we start with a new customer, we try to audit them all.
- [63:34]
A
Okay.
- [63:36]
B
We have team members, we have freelancers that helps us with that. We do it ourselves. As you might imagine, at 12:00am or 11:00am in the night, we are reviewing conversations many times. But after finding out that was super painful, we also have an agent that does a revision automatically and send us an email, an alert email, in case there's anything that needs to. That requires our attention.
- [64:06]
A
So this is part of onboarding a customer. You go through this testing phase to make sure the agent is interpreting the menu correctly. It knows how to construct an order. It's not something that you have to do indefinitely. It's just part of the, like, fine tuning.
- [64:23]
B
Just a few weeks.
- [64:25]
A
Correct.
- [64:26]
B
Just a few weeks of onboarding, and then it just starts on its own.
- [64:30]
D
Yeah, yeah. It used to be three months. And as we improve this, the time for onboarding is reducing, which is basically our biggest challenge. And what we're working on is like improving the onboarding times to make it super, super fast. Like, we think we've already solved the messaging part. We. I mean, even today when we are reviewing calls, messages with something, just so you know, the idea is like every noon or evening where like lunch or dinner time, Santi and I are sitting on our computers just watching these conversations going in, orders being placed and just seeing them go. And it's like we message each other going, did you just see what it. How it solved the problem? We even are amazed about how it's working. Right? So, yeah, it's impressive. So we think that part is solved. And the part that we're trying to solve right now is basically decreasing the onboarding times.
- [65:24]
B
When the type of restaurant is new, you might have a longer onboarding because of all of the different products. But for example, you can get us any pizzeria. And we will get it at pizza stores and we will get it set up pretty quickly because we know how that business works.
- [65:40]
A
That's it's cool to see, like, how you can build iterative domain knowledge and start to reduce that onboarding time for different types of businesses. This is great. It's really clear you're passionate about your problem space and that you've really dug in and that you have an equal passion for the technology, which is fun to see, too. Is there anything you wish I had asked you that I didn't?
- [66:09]
D
I think your questions were amazing. We got to talk about a lot of stuff.
- [66:16]
A
All right, then, let me ask you one last question before we wrap up. What's next? What's the big challenge that you're tackling next?
- [66:25]
B
Yeah, it's a great question and honestly, super proud of our product and we've seen it working in lots of venues. So now our goal is to scale it. We want to be, we want restaurants all over the world to be able to use this tool. We are focusing on Argentina, Mexico, USA and Spain. But we really believe that this, we're already seeing the numbers and it's impressive how much it helps businesses that that can use this type of tools. So next step, let's get it out there.
- [67:04]
A
Amazing. Hopefully this episode will help get it out there. Santi and Juan, it's been super delightful to hear about your story and to learn about your product. I look forward to when I get to actually order a meal through WhatsApp, so I will keep an eye out for it here in the US you will.
- [67:22]
B
Thanks so much for having us, Teresa. Super good question.
- [67:25]
D
Super, super nice.
- [67:28]
A
If you enjoyed this conversation, please subscribe in your favorite podcast app and give us a rating as it helps others find the show. Thanks. I appreciate it.