← Marketing Against The Grain
Loading summary
Transcript77 lines
- [00:00]
Kieran Flanagan
Okay. This year has been called the year of AI Agents. Now there's a ton of buzz about agents on YouTube, on X, all of these different platforms and actually a lot of it is wrong. A lot of these agents are not very good yet. We want to give you the actual real down low of what is happening with agents and we're doing that with one of the best brains in the business, Joe Mora, the CEO and founder of Crew AI. We're actually going to break down what is an agent. We're going to tell you where you can get started to build agents for role today and where they'll have impact. We're going to give you real use cases, things that agents are actually doing and make an impact for businesses today in marketing and sales and other places. And stay tuned because we are going to tell you your job in the future is going to depend on the quality of the agents you have built to help you do your role. All that and more on this episode of Marketing Gets the Grain. I'm your co host as always, Kieran Flanagan here, as always for my co host, Kip Bodnar. Let's get into today's show.
- [01:04]
Kip Bodnar
We right back to the show. But first a quick word from our sponsor. Remember when marketing was fun? When you had time to be creative and connect with your customers with HubSpot, marketing can be fun again. Turn one piece of content into everything you need, know which prospects are ready to buy and see all of your campaign results in one place. Plus, it's easy to use helping HubSpot customers double their leads in just 12 months. Which means you have more time to, you know, enjoy marketing again. Visit HubSpot.com to get started for free.
- [01:49]
Kieran Flanagan
Okay, we're here with Joe Mora, the CEO, founder of crewai. CREWAI is your global control pane for agents. One of the best minds on everything there is around agents. Joe, very happy that you're joining the show.
- [02:03]
Joe Mora
Hey there. Thank you so much for having me.
- [02:05]
Kieran Flanagan
So Joe, this is like the year of agents, right? This is all we've heard. I actually have been paying attention to a lot of the things happening this year. I think it was World Economic Forum and there was another big meet up of all the AI minds and I think the number one thing on everyone's lips is like agents, agents, agents, agents, agents.
- [02:22]
Joe Mora
Right?
- [02:22]
Kieran Flanagan
That's most of what we are all talking about. And so I thought we could just tee up for our audience. Maybe we'll actually start with what is an agent. Maybe you can explain to our audience like what do we mean by agent. You're one of the best minds here to explain agents. And we can actually get into your take on is it really the year of agents? And what do we even mean by that? Like, where do you really see agents being impactful this year? So maybe start with like, what is an agent? Like maybe explain to our audience how you think of it.
- [02:51]
Joe Mora
Well, first of all, thank you so much for having me. I'm so excited that we get to talk about this. And yes, everyone wants to talk agents. And I gotta say that that has been an interesting year so far, but that's a great way to start, right? What is an agent? What is not an agent? Like now you have operator, you can chat with all these things, like how you draw the line. So the way that we think about it is everyone knows about all these LLMs, so chatgpt, entropic and everything. So they're very good at kind of like predicting content. Right. So if you really say like, hey, write me an email, it will do it for you. And if you say, well, make it funnier, it will do that for you. Now the interesting thing is it almost has some sort of cognition. Right. If you give them two options of emails, it's going to choose between the two and give you a reason for that. So the beauty of agents is you can exploit that feature to how these LLMs kind of navigate a problem on their own. So it's not a chat anymore. You give it a task and you can leave the room. And then this agent's going to try to autonomously, kind of like through this idea of reasoning, figure out how to get there. So I would say that the definition of an agent is it got to have agency.
- [04:03]
Kieran Flanagan
Yeah. And it's got to have like some components. Right. Like it has tools, it has memory. Like maybe just talk about some of the common characteristics of what an agent will have.
- [04:13]
Joe Mora
Yeah, because if you think about the LLM on the silo, it just speed tests out. Right. But in order for you to make this more agentic, you need to have a way to hold that information. So you're going to need like some sort of memory. And then there are going to people that are going to ask about short term memory and long term memory and you can really dive into that and things get very technical. But there's memories and there's also tools. I would say those are the two big components. So as these agents are trying to do something, they're going to use those tools to interact with other systems. ERP or CRM whatever that might be.
- [04:46]
Kieran Flanagan
Right. Really, when you think about an agent, most of them will have memory because you need that to have, as you said, agent behavior. And most of them will have access to tools because they can actually do things on your behalf. They're autonomous. And so as we sit here today, one of the things that Kip and I were jamming on earlier on, and when I say jamming on, I mean Kip showing me because I'm a European and I live in the dark ages, and we are not allowed to have access to anything without the bureaucrats signing it off for us. And so I do not have access to ChatGPT operator today other than probably through a VPN at the weekend. But it's a good segue into like the year of the agents and how you think about that. So OpenAI launched that. I think that, Kip, your take was, it's cool for geeks like us. Why don't you give us your take, Kip, on the words in your mouth? And then I would love to get Joe your take on their launch and just where we are in agents in general in terms of capabilities.
- [05:39]
Kip Bodnar
Yeah. So if folks aren't familiar, OpenAI launched Operator as part of their ChatGPT Pro Edition, which is the $200 a month edition. So I immediately had to upgrade from $20 a month to $200 a month.
- [05:53]
Kieran Flanagan
They took your money?
- [05:54]
Kip Bodnar
They took my money. I want to know one pro and everything too. So I got to do that upgrade. So there's already a cost barrier. And what it does is it has basically a browser control agent where OpenAI has built their own browser, and the AI can go and navigate based on a request. So you can make hotel reservations, flight reservations, dinner reservations, do research. And it's kind of slow. It's kind of clunky if it's a good background task. And it doesn't require me to, like, log in or do a bunch of interaction. It's just like, hey, can you look up a hotel with a pool in the city? Like, that's great. It can do that real quick and give me all the information real slow.
- [06:35]
Kieran Flanagan
It doesn't do real quick. It doesn't much slower than, like, it's like 10 times slower than a human. You could have went to upwork, hired a freelancer to be your personal searcher, and had them search for it in the time that OpenAI's operator actually completed the task.
- [06:49]
Kip Bodnar
Look, it is very slow right now. I think it's a peek at what's to come. I'm Interested, Joe, in your take. But it's like, I think it's a look at the future, but it shows that the future is still a little ways away.
- [07:00]
Joe Mora
Well, I'm going to say I agree with you, but I have, I think would be a hot take, maybe not a hot take. I don't know. And that's how I started to build crew. Right. My first experience with crew was building agents that would help me with, like, posting things on socials. I was never really good at that. I had all these ideas. But from getting that idea and making that into a, well, draft thing that you feel comfortable putting out there in the world, like, there's a huge gap in there, right. And I could do it for sure. It's just a matter that you're going to have to sit down, you have to spend one hour kind of like doing this. And if you want to do it consistently, you're going to have to do it every day. And I was never able to. The minute that I got agents to help with that, I start doing it every day. Because you're right, they would take way longer. But now I could speed out my crazy idea and I would make a coffee, I would get to work something. I would forget that the thing was running and then I would go back and it was ready. I was like, all right, this is good. So I hear you. I think they're not optimal or faster than humans by large. There are ways that you can get there specifically if you fine tune certain models for certain things. But I still see a lot of value, especially in kind of like this idea of you're firing a bunch of them in the background and you got to do something else and just not.
- [08:19]
Kieran Flanagan
Think about it, as you said. I agree. What Kip said as well is the latency doesn't matter if it's a task. That is this task that you just want to put on in the background and you have this little army of agents doing things on your behalf.
- [08:30]
Kip Bodnar
So here's a good example of something I think Operator would be really good at. Let's say you're moving into a new apartment. You have a room and like, maybe you've picked out a bed or something. You can upload the picture of the room, you can upload a picture of the bed and you can have it go research stuff that would go with it and like put together a room for you. And then you'd basically have a bunch of links you can go through and decide if those things actually go together. And it just happens in the Background and that stuff that you would have taken you a bunch of time anyway that you now kind of have crossed off the list.
- [09:00]
Kieran Flanagan
Right. The long term goal to have a personal assistant, which is what they're all building towards. There's one other thing I want to just touch on here because it's important of agents in general. Joe, before we kind of get into just your take on what you think about the impact agents will make this year. And it actually is the ux pattern that OpenAI had in the operator. Because one of the big question marks around autonomous agents is what's the right UX pattern? So people feel comfortable with them. And what I mean by that is, you know, these agents are autonomous. So when Kip asked it to book a table on OpenTable, should it just come back and say it's booked or should you actually be able to like see the agent complete the task? So you feel comfortable with the agent doing something on your behalf and they went down the path to have a UX pattern where you can see the agent doing its work and at any point in time you can take control away from the agent. But what's your general take on autonomous agents in terms of how comfortable humans are going to be to integrate them into the workflow? Like the right kind of UX pattern?
- [09:56]
Joe Mora
Yeah, that's a great question. I actually spent some time talking with Sam Altman about it and it was pretty good. Because if you think about it, it's like AI is moving very fast, right? So it's not waiting for its native protocols. Because if you think about it, it should not even use a browser. Right? A browser doesn't make sense. The concept of the bottoms and everything, that doesn't make any sense. Or even keyboard and mouse, that just increases latency and reduce throughput for these models, like in matter of fact, they should not even use language to communicate with themselves or have their own language. But there's something beautiful about being able to inspect what is happening there. There's a security aspect of it because you can always see what is happening. Right. And I just think it's getting better and better. You want to make sure that you're able to do that, but there's also that ability for you to feel reassured about it. So I think it's going to come to a time where you're going to feel good about just firing requests into the void and things are going to get done for you. But I think right now people just wanted to feel they're more in control of these things. Right?
- [10:55]
Kieran Flanagan
Right.
- [10:56]
Joe Mora
And this is true not only on the personal level, but also on companies, if companies are thinking about. And this we're seeing firsthanded on our enterprise deals. If a company wants to automate a critical part of their process, they want to make sure that they can visualize and control this and they understand what is happening and they can audit it later on. So I think it's a temporary thing that might change in the future, but I just don't know yet.
- [11:20]
Kieran Flanagan
Yeah, I was talking to someone who had been using very early on when agents first come out, they are playing with all of the technology. And they were set up an autonomous agent to do some stuff on email. And the agent was meant to craft emails, put them in the draft, and then they would go in and look at them and then send them or not send them. And they had realized that the agent started sending them. They were sending some like pretty bizarre emails and they were like, yeah, like I feel the right UX pattern is I need to be able to see what the agent's doing. And so maybe that's a good segue to like, you know, there's a lot of, hey, this is transformational. This year agents are going to be part of the workforce. That was the common talk thread over the past couple of weeks from every tech leader there is. What do they mean by that? Like, when you look at what's happening with agents, what do you think they mean by that? And what are you bullish on and what are you not bullish on when it comes to agents?
- [12:09]
Joe Mora
Yeah, that's a good one. Well, first of all, I think I want to show you a visual real quick because that will help me to make my point. And that is agents are happening. Right. So in here, what you're seeing is the number of crews executions per month. This was pulled a few days ago. So January is still going and up to this point has been over 16 million crews.
- [12:33]
Kieran Flanagan
Wow.
- [12:33]
Joe Mora
Each crew has many agents within it. Some have up to 21. That's the highest number that I have seen, but you can go higher than that. So what we're talking here is tens of millions of agents being executed every month. And the reason why I want to double click on that is just that it is happening. In my mind, it is happening. The genie is not getting back into the bottom now. It's a matter of how fast it will happen and how good you get in what period of time. But I think what a lot of what we're seeing on companies being bullish about it is because. And I'm going to take one step back. A lot of people are comparing agents with the Internet early days, and I think that's an interesting way to correlate the both. But if you were online on the Internet on day zero, you would have no upside, no impact on your bottom line because no one was online. But what we're seeing with AI is companies implementing it in like 2/4 later, they're reporting impacts on their bottom line. So you got some of those 10Ks and 10Qs and you see companies like Walmart starting to save millions of support and you're like, all right, same thing is happening here. So I think there's a mandate on the executive level, on these companies, on the board level of these companies, that this is happening three years from now. We need to figure it out how we're going to manage, deploy these resources. And from the edge side, again, everyone's tinkering with it and excited. So it's almost like a claw motion where like incentives are aligning. And I think that is just fooling this up. So I'm bullish that this is the year of agents. And what I mean by that is people are going to deploy a lot of agents this year. They're going to try a lot of things this year. Now, is this the year where companies are going to automate entire departments? I don't think just yet, but I think this year is where things start to get very pretty.
- [14:27]
Kieran Flanagan
Seriously, could you maybe tell us how companies that are deploying these crews and agents are doing it in the right way? So to your point, the wrong way is probably I'm going to go in and create agents to replicate what these humans are doing versus a lot of the success I have seen in agents is like they complete micro tasks that make up part of your role, freeing you up to actually spend time on things that are much more important. Right. So actually the human can become much better at that role. But could you maybe talk to us? Where do you see companies deploying them in the right way and what examples kind of use cases are you seeing that work really well?
- [15:08]
Joe Mora
Yeah. So I actually spent a good time talking with Jacob Wilson. He's the commercial CTO at UWC and Genai. And it's amazing to work with them. They're using crew a lot. And one big thing that we talked about, and there's a whole interview I can send a link over to you for, if people want to watch, is there is a cultural aspect on adopting agents in a company. Right. People fear that or people Trying to understand what role they will play. The companies that are being most successful, and PwC is one of them, what they are doing is they're kind of promoting people. Right. So you're still accountable for the end of Rezu, you're still accountable for reviewing this, put a nice bow on it, present this. But now you have this extra tool and these agents can do something for you. So a cool example that I can mention is we are working with a telecom company on a legal use case and basically they have their legal people that can do everything. But what is happening now is they're automating a lot of the contract analysis with these agents beforehand. So by the time that it gets to legal, it already has recommendations, red lines and a bunch of other things. So kind of like basically scaling things that they couldn't do before. We found it being prohibitive, expensive. But there are so many more use cases and we can talk about the sales and the marketing use cases and the back office automation. There's quite a lot going on out there.
- [16:34]
Kip Bodnar
We've talked a lot about kind of where agents are. And now I was trying to understand like what are the core use cases that people are actually building? What should people go and do? There are a lot of people who watch our show who are like, hey, I just want to like understand what I should be doing, how I go out and build an MVP of that to see if it actually makes sense for me, my business, what have you, like what's happening and where should people start?
- [16:56]
Joe Mora
Yeah. So what we are seeing is one, and I'm going to share a visual. I think this will help people see it as well. These are the most common horizontals within a company that we are seeing agents and use cases being deployed. Now there's a few interesting things that you can infer from this. One is there's no clear winner. Right. There's no like no, oh, people are using for marketing alone. No, it's very much spread out. What for companies like us is good news because means that you can land and expand into other areas. Right. But if you interview these people as we did and we talk closer with them, the common pattern is actually starting with simpler use cases. It's what we call low precision versus high precision. So low precision use cases, they require let's say 9% certainty or accuracy on their outputs. But high precision use cases require 99.999. So an example of a low precision could be, well, I want agents that will help me draft presentations for sales calls out of cross transcripts or my CRM information and high precision use cases that we are seeing out there is helping companies and banks fill IRS forms where like you don't want to get that wrong. Right?
- [18:12]
Kip Bodnar
Yeah, you can't mess up your tax.
- [18:13]
Joe Mora
Forms, especially like if we're talking about a big corporation. Just an anecdote. Funny enough, some of these forms, they are 70 plus pages long, but they come with an instruction manual that is 620 pages long. So yeah, agents can help with that. But that's more high precision use cases. Right. You're going to don't start there, you might get burned. Start with low precision and scale from there.
- [18:39]
Kieran Flanagan
That's really great advice actually. I just want to kind of recap that for the audience. So if you were like in a role and you wanted to even start with agents before you've even started to get into how to build them, and we can get into that and show you some use cases, what your recommendation is, is like look, and AI can help with this because I've actually used it as a test for this. If you actually take a role, let's say your role is a BDR and you split out your tasks into low precision, high precision and you can actually start to pick someones that are in that low precision category. And that's like some places to experiment with agents. Is that like the right way for someone to get started?
- [19:22]
Joe Mora
Yeah, the way that we put it is basically you have four bullet points. One is be another adopter, you want to get ahead. Two is don't wait for other people use cases. Right. A lot of people want to know what is Company X doing? Like you don't want to go there. So start simple. That's basically what you're saying. Start with something simple. And when you expand, you want to expand into kind of like the low risk, high impact, kind of like use cases and go in that direction. But yeah, exactly as you said. And then in order for you to build them, there's so many different ways right now. Right. And if anything there's going to be even more for less technical people. There's like no code platforms and we offer that as well. For example career AI. But there's many others. Like if you're a more technical person, you can use frameworks like Crewai itself where you can actually code in Python, some of these agents and some of this prompt. So I think there is a bunch of different flavors that people can use now. A lot of the true value gets unlocked when you get more technical people involved. I would say you Got that right. And that is you start with kind of like low precision use cases and sample and then expand spend into low risk, high impact.
- [20:39]
Kieran Flanagan
Right.
- [20:45]
Kip Bodnar
Let me tell you about a great podcast. It's called Creators of Brands. It's hosted by Tom Boyd. It's brought to you by the HubSpot Podcast Network. Creators of Brands explores how storytellers are building brands online. From the mindsets to the tactics to the business side. They break down what's working so you can apply that to your own goals. Tom just did a great episode about social media growth called 3K to 45K on Instagram in one year, selling digital products and quitting his job to go full time creator with Gannon Mayer. Listen to Creators are brands wherever you get your podcast.
- [21:24]
Kieran Flanagan
So maybe we want to get into like show some of these use cases. But I do think an important point to make is, okay, I've decided that I understand what an agent is. I've looked at my role and I understand like what are good use cases for an agent. The other thing you mentioned is like how agents are being adopted within the company, like how they're being built for different teams. Can you maybe just talk a little bit about that trend? Like how you see companies adopting agents.
- [21:56]
Joe Mora
Yeah.
- [21:57]
Kieran Flanagan
And you mentioned it's like still quite a technical thing and maybe kind of just touch on that.
- [22:02]
Joe Mora
Yeah, for sure. I think it's interesting because early days, what you would have with like LLMs, and I think a lot of people probably experiences this firsthanded is you would have on the edges of the organization, on the individual teams, people would just pick up LLMs to do work for them. So someone that's a little more savvy would start to play around with ChatGPT and they would found these amazing prompts that would help them with kind of like customer proposals. Right. And then that became a problem because, well, this person is putting information that might not be supposed to be there, the company doesn't approve. And maybe this person unlocked an amazing case. But that information now and that knowledge is siloed. So what we are seeing with agents, at least on the enterprise sales motion for creature AI, is more of a central deployment. So usually under a cio, a cto, a head of AI, and you sell into this department that is going to configure everything and then you start to enable these departments individually. And the cool thing about that is you're going to have way more control on what are the LLMs that are being used. Do I want to add filters for PII and personal information and making sure that I'm controlling all these use cases so they are reusable and then even enabling people that can code to use a platform to build it. So that's kind of like what we're seeing in terms of the enterprise adoption. And again, funny enough, there's a lot of kind of low code, easy to roll out and templates that you can use, but once that you want to get into those very kind of like chunky use cases. Right. Because if you forget the name AI agents for a second, we are talking about AI powered automations. And automation is going to be a very kind of like customized process because all these companies have no eco process. It's when technical people and being involved unlocks a lot of value, especially on integrations with homegrown systems and such, right?
- [24:03]
Kieran Flanagan
Yeah. Like they're much more powerful if they're deeply intertwined within your unstructured data and they're deeply intertwined within your systems.
- [24:11]
Joe Mora
Yeah. But I gotta say, it's still very much early day. So we basically pulled around 4,500 people are from different companies and only 15% actually have features in production. Right now if you isolate at this only looking at large enterprises, then we're talking about 23. So it's a higher number. What kind of hints that enterprises are moving a little faster here. But for a lot of people it's in the early days or they just have done a few deep dives. So I think this is very interesting to watch.
- [24:42]
Kieran Flanagan
What is that, Joe? The 15% is enterprise companies that have used an agent to complete a task or what's the 15%?
- [24:50]
Joe Mora
No, that 15% is overall companies, not only enterprises that have features with AI agents in production. And if you cut off and look at only the enterprise, that number bumps into 23%. Okay, but that is having like AI agent powered features in production from the companies that we talked with, not necessarily in their product. Might be like internal automations. Right. Might be a backups automation.
- [25:15]
Kieran Flanagan
Okay, so it does internal.
- [25:16]
Joe Mora
Yeah, I think internal is actually this is one thing that I found very funny. I mean, I had this hypothesis in 2024 that I would see way more SaaS companies kind of like adopting agents, just kind of like trying to reinvent themselves. But funny enough, I'm not seeing a lot of that. It's a lot of more traditional organizations that are kind of like trying to figure out how to get more efficient. In the SaaS companies that we have talked with, I get more of reluctance from them on it and not sure what that's coming from, but it's definitely one hypothesis that that have been proving wrong.
- [25:47]
Kip Bodnar
I feel like that number is higher than I would have expected. I would have expected like single digit adoption of like agents being deployed. It just, it shows that even though it's early, there is a lot of just crappy stuff that has to get done out in the world that agents are good enough to go and solve today. Right. Or otherwise that number would be much, much lower than it is in terms of like agents deployed.
- [26:09]
Joe Mora
What I would say though is there's one caveat, right? These people, they were coming into CREWAI to answer this form, so they are definitely more savvy, they are looking agents. So I think if you look at the broader population of like any company out there, you're gonna have like smaller numbers. Like these are companies that are actually engaging with us.
- [26:26]
Kieran Flanagan
Yeah. So I guess like even among the tech savvy, that number still looks low. But if you broaden out to like the laggards, we're still so early. I think the point here is like, yeah, if you're listening to the podcast and you're following along and you're thinking, well, I'm gonna create an agent and do something, you're in the fast movers.
- [26:44]
Joe Mora
Yes, exactly.
- [26:46]
Kieran Flanagan
I think one of the things that would really help our audience is showcase some of the agents that have been built for real use cases and go through like what these agents are actually doing for maybe a salesperson or a marketer or something that you really think is a good example of an agent in action.
- [27:02]
Joe Mora
Yes. So I think on the marketing and sales side, one of the most interesting ones that I have seen is agents that are doing a couple of things. They start with enrichment. So the way that they deployed this was actually an in product feature and a back office kind of like automation. So when someone would come into their website and create an account, they would have agents go online and start researching this person. So up to this point that's okay, right? This is kind of like regular enrichment. Like find what is this person role, find like more about this company vertical and all that. So that's good. Now where things starts to get interesting is they got some agents to get one step ahead and given the information that they found, come up with hypothesis on how this person is going to use their product. So what is the interest if this person is kind of like a CMO at a company, how they are going to use what value they're going to get from this product and create three hypothesis and then do the same thing for the company. For a company in this vertical, what would be kind of like the main three things that would take. So now that the agents have done that, they convert it into JSON, so structured data, and then push it in two places to their HubSpot and into their product database. So what that means is now out there, email marketing is like super hyper targeted, mentioning not only the name of the person, the company, but like highlighting the features and the ways that they could leverage. And then in the product, and this is kind of like what they're working on in the product. They preview some of the informations and the templates that they show, like basically using that inference that they made. So very interesting use case. And I think a great kind of use case for agent in general.
- [28:54]
Kieran Flanagan
Yeah, yeah. Somewhat similar to what we do. And I think the example going back all the way to what an agent is and having memory tools, like an example of a tool that is really good for. For research is something like Perplexity's new sonar. Like, Perplexity is a pretty good research tool that the agent can have access to and do some of that research on your behalf. So I think that's a good example of like the agent being able to autonomously complete those tasks without any real human in the loop. And you as a sales rep have that stuff at hand. And so you can be much more productive because you have this stuff being done for you by a small team of agents who each are specialized in one of those tasks.
- [29:33]
Joe Mora
Exactly. And I think we go back to the comparison that we make with the operator early on. Right. Like, yeah, as you as a sales rep, you could go out and research this customer and make sure that you're engaging them in a very custom format or prepare for every meeting that you have. Yes, you absolutely can. And you might be able to do it faster than what an agent would do. But if you can do that at scale, that means that you can take more meetings and be better prepared to them. And there's also something beauty about you being able to customize it. Right. Let's say that you always have that one sales rep in your company that is just like the beast, Right. The guy does the best prep. Why don't you just try to replicate that for everyone? And now everyone has that level of notes. I mean, the sales rep might not be too happy about it, but it might be definitely an interesting experiment.
- [30:18]
Kieran Flanagan
I would love to get into another use case you mentioned of Mike, but I think one of the interesting things here because it was a point you made, which is, you know, the rep can have these agents and be much better at their role and other reps might not be happy about it. Like, it creates some competition to actually use agents because you can't compete unless you have access to the same sort of help. And I don't know if you saw, there was an interview from Satya Nadala, the Microsoft CEO this week and one of the things he said was in the future people might be hired because of the agents that they have helping them do their role. And so he can imagine a world where you go to LinkedIn and instead of having certifications or anything like that, and even prior work experience, you actually list out the agents that you have built to help you be like the kind of employee that you are. Let's just get your kind of thoughts on that as a future version of what like a great employee might look like.
- [31:11]
Joe Mora
I 1000% agree, honestly, not because of the agent themselves, but also because what they tell about that person. And I can tell that we are doing this at career AI. So for example, when we interview people for engineering roles, we tell them during the interview you are allowed to use anything. Like you can use ChatGPT, you can use Entropic, you can use Courser, you can search Google, whatever it is you're allowed to use. If they don't use it, it's an automatic pass. And how well they use it actually counts a lot on if we're going to make them an offer or not. And that can be counterintuitive for a lot of people. Like, well, but are you assessing their engineering skills? Well, on the day to day they're going to have access to the tools anyway. So I want to know how well they use it. And I would say probably what he's hinting there is some of that as well. It's not about the agent that they have as the companion, but also the ability that they're able to create something like that and what they tell about that person.
- [32:10]
Kieran Flanagan
Yeah, coming back to the use cases you mentioned off Mike, I think one of the ones we should actually just cover real quick because everyone loves a good SEO content use case. Could you maybe just go through that use case? You just give another example of what agents can help with?
- [32:24]
Joe Mora
Yes, I love that one. That was so good. So this was a startup that we were helping on using agents. It was very early days in crew, it was very interesting. And what they wanted to do is everyone likes to do not only SEO, but overall conversion rates right And a B testing is such a big thing and everyone know that it works, but takes a lot of work to do it well. So what we're building together was agents that would like given your product, it goes into our website, would take screenshots of our website, understand the copies, everything, then would research what are their competitors. We'd go into their websites and look at all their copies and everything. And then we'd create hypothesis. What should we change on your website? Given what the agents were able to build an understanding around the industry and the competitors in order to get you a better conversion. And then the idea is that they would go all the way to implementing those A B tests and measure it so that you can then choose. So it's basically automating the whole kind of AB testing kind of process from SEO to copy and everything, but going this one step beyond kind of understanding the industry and the competitors and everything. And again, something that you could do yourself, but that would take quite a lot of time.
- [33:43]
Kieran Flanagan
So it's basically giving you recommendations on what a B test to run by ingesting that data and actually doing the AB test themselves. It's actually going ahead and just. So you just basically come in and updating the code?
- [33:55]
Joe Mora
Yeah, yeah.
- [33:56]
Kieran Flanagan
Explain to me what the agent does. Do you have to start off by giving it data? Like what's the human doing and what's the agent doing?
- [34:02]
Joe Mora
So the input there was, this is my website, this is my description of my industry, this is what I'm trying to optimize. This is the human input.
- [34:10]
Kieran Flanagan
Okay.
- [34:11]
Joe Mora
And then the agents would go around and do everything behind the scenes and they would come back with all this like AB hypothesis for testing.
- [34:18]
Kip Bodnar
That's pretty awesome.
- [34:19]
Joe Mora
And then what the company that we were working with was actually doing is building a product around this. Right. So you would be able to see how the hypothesis and you would say, yes, let's do this one, this one, this one. And then they would run for a few days and you could implement it. But that was like, that was not the agents anymore. That was the proper product.
- [34:37]
Kieran Flanagan
Did they fine tune the agent or use RAG or anything to build a hypothesis like, you know, or was it just the public information? The agents knows everything about the Internet.
- [34:45]
Joe Mora
Public information?
- [34:46]
Kieran Flanagan
Yeah. Okay.
- [34:47]
Joe Mora
Yeah. I think there was something about they're doing with the images that was kind of like more proprietary to them, like the parsing the images where they're doing something fancy with there's not only the screenshot, but there's also the HTML. And then they would Create a better understanding of like the web page by doing that. But no, that was basically a lot of kind of like the open models that you have now.
- [35:08]
Kieran Flanagan
Very cool. This has been a great conversation. I wonder if where we should end is. Okay, we started with, you know, explaining the agent how to get started, how they're going to adopt it, but really like framing it all. And like, we do believe this is your agent. You have some, like, really great stats to show that this is really happening. Right. This is not all hype, but what do you think is hype in terms of what you hear and how agents are spoken about? What is the mismatch you see today and where the technology is today and where expectations may be for how people want to use agents? Like, where do you see the biggest mismatch when you speak to people?
- [35:43]
Joe Mora
Yeah, I think honestly there's going to be a lot more humans in the loop than a lot of people believe. Especially this year. I don't think it's going to be kind of like once and done. Like we are seeing this with operator, right? And like now there's a bunch of tweets where like people set it to do things and kind of like fails midway through and you got to step in and take over and all that. So I think this is not the years where we're going to have like completely end to end, especially on the more high precision kind of processes. Kind of like, oh, agents are doing everything. I think that will be kind of like a step by step. The other thing is implementing on especially the more complex use cases is going to be a way, like a bigger lift than what I think, while other people would believe, I think there's a lot of glue in this company nowadays and in the softwares and how they connect to each other, that in order for agents to kind of like be able to navigate those paths, you're going to need to have like clear code and clear instructions on how to do it. And I think, like, there's a reason why everyone's doing the browser side of things first. Right. That is easier. Like there's a common interface, but there's a lot of companies out there, they have software that is not even online use, desktop apps, how you handle that. So I think there's going to be a lot of more challenges in there. So I think this year is going to be definitely where we're going to see a lot of agents going into production. But it's very much early days still. This is not the years, like where agents are taking over a workforce. That's not it. Right.
- [37:11]
Kieran Flanagan
And so the YouTubers creating hype, all those YouTubers, as we realize as YouTubers ourselves, everything has to be like a dramatic headline. That's the way it works. Joe, I think this was an incredible run through of agents in a way that will make it really easy for people to actually understand what the reality is and where they can go get started. And obviously, we would highly recommend they go to a platform like crew and build some agents and play with the technology. And you have a bunch of cool templates, actually, that make it super easy to start to get inspiration. One of the things that people actually struggle with, which is why I wanted to really dig into, like, how you would suggest someone starts with a use case, is people just struggle with, like, where do I even get started? And I think your template gallery is a really great way to find inspiration for your role. So I appreciate you coming on and taking us through that explanation of agents.
- [38:04]
Joe Mora
No worries. Thank you so much for having me. I had a blast. Thank you so much. And I catch you online.
- [38:09]
Kieran Flanagan
Thank you, Joe.
- [38:09]
Kip Bodnar
Appreciate your help. Ra.