Loading summary
Transcript80 lines
- [00:00]
A
I know it's crazy, but there are people making a million dollars a year. As for deploy engineers, but what exactly is an fde? I know you've probably seen it, but I feel like a lot of people aren't clear as to what it is and how they could become one. Well, in this episode, I brought on my friend Vaas. And Vaas is a leading expert when it comes to fdes with his company, Veric Agents. And in this episode, he gives you his entire playbook, how you could become in FDE in 30 days. Now, this episode is for people who want to become an fde, but also for people who are just interested in what it means, how they can actually use fdes in their business to make more money, to be more productive. And I just think this is the clearest episode, the clearest piece of content on the Internet, the clearest masterclass for how to. To understand clearly what an FDE is, how you could become one, and why it matters. Enjoy the episode and I'll see you at the end. Vaas is here. And Vaas, by the end of this episode, what are people going to learn?
- [01:20]
B
They're going to learn exactly how to break into Forward Deployed Engineering or become a better FTE in 30 days, the full roadmap for AI forward deployed engineering.
- [01:29]
A
And I don't feel like this has been shared anywhere. I feel like the term for Deploy Engineer is just like on my X feed everywhere. So what I'm hoping for, vas, is for you to clearly explain what this means and just like all the concepts to it and just break it down for me in a clear, easy to understand way. So. So I could. So I could, you know, learn from it, but so others can learn from it too?
- [01:55]
B
Absolutely. And there's a lot of different definitions. Everyone has their own. And I'm going to give you what I think is the clearest explanation.
- [02:01]
A
Okay, let's do it.
- [02:03]
B
Sweet. So, yeah, this has never been shared before. This is something our team put together. It's how to break into FDE in 30 days. Let's start off with recent developments in the facts of today. The reality is every company can now buy intelligence. You have a frontier model being released every single day. Just Yesterday we had Kimme3 being released. Last week was Fable5 or GPT 5.6 SOL. Every company can now buy intelligence. And the reality is intelligence is becoming commoditized. So the same foundational capability is becoming available to anybody who can pay for it. And that's most companies today. So you'll see Here is a graphic where every company has access to the same foundational model. So if everyone can access it, intelligence can no longer be the mode. I think there was this huge theory that people will be priced out of intelligence, and that could be the case in the future. But the reality is today everyone has access to the same tools. If you go and talk to 50 different enterprise clients, they're all using the same stack, they're all using Claude Code Codex, they're using Cursor for model agnosticism, they're using GitHub, Copilot, it's all the same thing. So the reality is everyone has the same capability in terms of what intelligence tap they have access to. So where does the advantage go? It goes into deployment. So the edge is no longer who has the intelligence, it's where, how and why they use it. And that is the role of an AI forward deployed engineer. It's allowing companies to harness and take advantage of AI intelligence or software previously to make sure that they apply it the best to their specific company context. Every single company is different in terms of how their business is structured, what processes they have, how things run, etc. And the job of an FD is to make sure that the intelligence, which is general, is specifically applied to this company in a way that benefits them the most. And the advantage will start to become who has that best bridge that connection between their own processes and the intelligence stack that they have access to.
- [04:10]
A
And vas, this was a term that, correct me if I'm wrong, was popularized by the Palantir team, right?
- [04:17]
B
That's right.
- [04:18]
A
So can you tell me about how Palantir. I think how Palantir works with FDEs. I mean, I feel like for a lot of people, Palantir is this black box. Can you go a little more into that?
- [04:30]
B
Yeah, it's funny. So I lived in New York for a few years and I had a ton of buddies who are Palantir for deployed engineers. So I have a little bit of insight into this without sharing what I think is proprietary. Palantir has an ontology. They have a software stack that they work off of. And this is full of connectors to software, but also data lakes which allow people to. Which allow enterprises to pipe their data into a unified interface. And what Palantir FTEs then do is they'll actually be deployed on site. So this is either like enterprise customers or the military, the government, and learn their workflows and then spin up kind of workflows, dashboards, you know, agents that will solve the problem. For these companies. So Palantir really popularized the idea of essentially which was what was consulting but for the software space. They coined the term for deployment engineer and it really started to allow their business to take off. They had a centralized platform, but its beauty was not in like how tech forward it was. It was, it was in how customizable it was. And then these four deployed engineers would go on site, customize it for a client, and it would really solve their pain points better than a generalized servers. Cool.
- [05:37]
A
And I guess like the thesis is that if it works for Palantir, it can work for everyone else, right?
- [05:46]
B
That is sort of the thesis. Yeah. And the Palantir I think really solved it when it came to the data age where you wanted to unify your data sources and then visualize it in more unique ways. I think the AI age is going to demand that 100 times more where every single company is going to need customized agents. And that's actually what we're here to solve as well with our os. But everyone is sort of coming to the same realization that forward deployed engineers are a massive reason why AI is going to be powerful for businesses.
- [06:14]
A
Cool. Let's keep going.
- [06:16]
B
Sweet. So someone has to decide where intelligence belongs, not just where it's applied. And that person is a forward deployed engineer as well. So there's kind of three stages to a forward deployed engineer's involvement at a company. The first is understanding the business reality. So it's how the work actually happens today. And I think this is the part that gets glossed over by people who are very deeply in tech. In Silicon Valley we have a mindset where like, okay, well the beauties in the software, it doesn't really matter what the business processes are. But I can tell you firsthand, every business is so different, even in terms of the same process across multiple businesses. Let's go accounts payable for example, or sales for example, at two different companies. The way that they do that is so different. One has a ten step process, one has a thirty step process. One is using Salesforce, Gong and Chili Piper. The other one is using HubSpot and Apollo and Clay. There's the software differences, but there's also the process differences and there's the things that matter most of the business being very different when things go wrong, like exception handling. The way that that's being done at a company needs to be documented as well. So forward deployed engineers go on site, they'll either interview people or just observe them work or get access to their systems like their ERPs. Their CRMs to figure this stuff out. And this is where the bulk of the time goes, in my opinion. It cannot be understated how important this is both in terms of understanding the business, but also to bring that business along the journey. And this is where communication and analytical ability is incredibly important for a forward deployed engineer. The way I like to think of an FTE is the best combination of someone very deeply technical who can understand it, but also someone with fantastic communication ability and the ability to get the information out of people that they need to. And this is all encompassing for business reality.
- [08:14]
A
And you know, you say the FDE goes on site. When you say on site, is that like literally like walks the office and starts like speaking to people and stuff like that or do you mean like,
- [08:25]
B
you know, it can, it can certainly be done remotely.
- [08:29]
A
Yeah.
- [08:30]
B
And you know, there's, there's certain times where that's, you know, required because the company itself is remote or like the people of the function are not all in one office. But I will say a large majority of the time it is on site. I know that Palantir does this very heavily. We do this as well. And it's not just because, you know, you can't get the information remotely, but it's actually more so because the relationship that you build with the person, like, you know, you're on site, you're part of the team, you'll uncover far more information that way. Right. Like if somebody will, if you schedule a one hour meeting, for example, somebody will walk you through what they think is the job. But if you're on site with them for the full 8, 10 hours, whatever it is, you're actually experiencing the job. Like when something goes wrong that's not really documented in an SOP or like a, in a word Doc or Google Doc somewhere, you're going to see that play out. You know, Even consultants at McKinsey, they do this, they'll go on site to a mine and they'll sit with the miners and they'll watch them do their work because it's so much more powerful to establish that relationship and get the information that way.
- [09:32]
A
Cool.
- [09:34]
B
Yeah. The second step is FTE judgment, which is where does intelligence belong and where does it not? I think when the AI wave first started, we saw a lot of let's just slap AI everywhere. Let's give everything to the model and let's let it figure it out. And this is what led to token maxing and hallucinations. And you have the mit stat that 95% of generative AI pilots fail again. Now the industry is sort of shifting gears and they're realizing that, okay, we actually need to be very selective about where we apply this intelligence. And we also need to be selective about how we design the new stack for this intelligence to play out. So again, for example, you have a ten step workflow. It might be that that workflow should not be changed by AI. Maybe it's too risky or maybe it's not high enough roi. Maybe it's already pretty automated. Why do we need to bring AI into also could be that of those 10 steps, only three of them actually need judgment. Right. So categorization of this lead in a CRM tool that might be a little bit more non deterministic. So we're going to bring in an LLM there for judgment, but the rest can be solved with if then else statements. It can be solved with API calls. And this sort of judgment is actually far more complex than I'm making it out to be even, but it belongs with the fde. So the FD both has again the business reality, which is the consulting style, like communication style approach, but also the technical judgment so that they can determine based on their technical background. Okay, I think that this design is going to be risky. We're going to have 80% accuracy, it's not worth it, versus this other workflow will have a much higher ROI, we'll be able to build it much faster, it's lower risk, etc. And that sort of back and forth judgment is where an FTE really shines. It's bridging the gap between the business and the technology.
- [11:27]
A
If you, you know, just curious, like if, if someone listening to this like wanted to become a Ford deployed engineer in New York City, that's doing this sort of stuff, like how much money could they make?
- [11:40]
B
A lot of money. You have no idea how expensive it's gotten. Both from a we're hiring perspective, but also in terms of what the market's demanding. This is the hottest role in technology right now. I mean you could make anywhere from 150,000 base with considerable equity to. I've seen roles go up as high as a million dollars a year. And I'm not joking. These are extremely well compensated roles. If you are the best combination of consulting and technology.
- [12:07]
A
Cool.
- [12:10]
B
Deployed AI system. Finally, the last step is actually going out and building the software itself. This is where it varies wildly, company by company. For example, at Palantir even they have some FTEs that, you know, you're not actually writing code, you're Mostly like spinning up workflows like text, you're chatting with the software, the Palantir ontology, to create, like, some dashboards. And to the extent that you're writing code, it's SQL. But there's other companies where you are fully writing, like, production code, either on site with the client or you'll go back and do this. But it varies wildly. So there are some FTE roles where you're going to be writing production code. You need to have a background in software engineering and really be confident in your ability there. There's other FTE roles where it's far more technically light and you can chat to build on top of an existing platform. So this is the part that varies wildly. But either way, you need to have a very good understanding of the software, because when the client has an issue with, hey, this doesn't work right, or we have an issue in production, it's basically your ass on the line and you have to know who to call, what to do and how to fix it.
- [13:14]
A
Totally.
- [13:15]
B
In summary, fdes are in demand because they control how intelligence enters the business, how it's used, and that is where all the value is today in the AI age. Everyone is coming to this consensus, and that's why FDs are extremely valuable.
- [13:28]
A
Well, it's also in demand because it's new as well, like this. Like, there wasn't Intell super intelligence on tap five years ago. So not only is the idea of superintelligence on tap just absolutely absurd, many trillion dollars of money's gonna be changing hands over the next few years. But the idea that now you need a person to actually, like, people are realizing, hey, you actually need judgment. And, hey, you actually need to, like, be a systems thinker. And, hey, like, actually, token maxing isn't the best strategy. Like, there was, like, a moment in time where token maxing, like, people were. People basically were agreeing that token maxing was the strategy. It was just like, hey, these models are so good. Let's just let them do their thing. That was, like, the thinking.
- [14:21]
B
Yeah, that was a funny period in time. I would argue we're still not fully out of that, but, yeah, you're totally right. I mean, this is a brave new world for everyone involved. And, I mean, I have horror stories of C Suite executives I've talked to who have blown through their entire $10 million cloud budget in, like, three months. It was supposed to last them a year because they gave it to everybody. It's token maxing, and everyone's spinning up whatever they need and the sad reality is it didn't really move the needle for the business either. And it's because that business didn't really invest heavily into forward deployment engineering. And so I think you're totally right.
- [14:57]
A
Cool, let's keep going.
- [15:00]
B
Sweet. So we've already alluded to this pretty heavily prior, but I want to really make sure I hammer down this point, which is that there's two sort of streams of kinds of judgment that are required and it's very rare in a single person. So I think the unfortunate reality is, and this is what's going to happen, it's already starting to happen, is as we go from the token maxing, let's go all in on token maxing. Go, go. To now the same thing on FTE. Let's go, go, go. Let's hire a bunch of FTEs. I want to be very clear about what the role really demands. I think there's a lot of FTEs who are unfortunately neither the best communicators and neither the best software engineers. I would strongly urge them to strengthen both of those skills. It's both the understanding of workflows, cost incentives, risk adoption, business value, the politics of the internals of a company. These are all things that you have to manage. And consultants here are incredibly strong. Right. You'll talk to McKinsey Engagement Managers, BCG, Bain Engagement Managers. They'll be very good at this side. The other side is what they might need some more support in. And same thing, software engineers will be very good at the right side with models, systems, APIs, data code reliability, evals, guardrails, you know, more AI centered turns, harnesses, post training, fine tuning. These are things that are more on the technical side of the aisle. And software is usually very good at this, but they need to also then kind of drift towards the business side by understanding the left side of the aisle. And FTE is the best combination of both of these. It's not an average combination, it's not the worst combination of both. Where you're not the best communicator but you also can't code. It is truly the best of both. And that is the million dollar hire where the FD can turn business understanding into working software end to end. They can do both sides perfectly.
- [16:47]
A
Yeah. I mean, put another way, it's like if you understand art and you understand science and you could speak both,
- [16:56]
B
you
- [16:56]
A
have what it takes to become the million dollar fd. The hard part is like usually the people that are good at science are sort of good at science, and the people that are good at art, are kind of good at art. But there is some overlap in the Venn diagram.
- [17:15]
B
Absolutely. That's why it's such a rare role. But I also firmly believe, and that's kind of the whole point of this presentation is that you can become this. It's not out of the realm of possibility to become much better at both of these things. It just, you need it cleanly laid out. You need a roadmap. And that's what I hope that I can provide by the end of this call. Cool.
- [17:33]
A
All right. All right, Vaas, let's go.
- [17:38]
B
So first, you know, for example, let's understand how the work is really done, because the documented process is very rarely the real process, Right? So an email might arrive. Now, this is extremely simple, but the reality is it sounds like a clean trigger, but it's far more complicated than that. It arrives from 40 plus senders. No two of them are formatted alike. The data is different. Some of it's in a PDF, some of it's in a screenshot, some of it's in an Excel spreadsheet or it's buried in a forwarded thread. It's far more complex than it makes out to be. So if you didn't look into this, if you weren't an FD and you just asked the person for, hey, what's the first step of the workflow? They'll tell you an email arrives and all of a sudden you're building for a system that doesn't map to reality versus the reality, which is that it's so complicated. And half of them are exceptions. It's the same as last time. It ignores the second attachment. Sarah already signed off on this one. You. There's no consistent subject line, so you can't route without actually going into it. And usually the reality of how to play this is in one person's head. So one person will know, like, okay, yeah, well, when I see this email from this person, I'll send it to this, this vendor or to this part of our procurement team. But that's not written down. And if you don't sit with that person, kind of coax this all out of them, they're not going to remember to even tell you. You would think about, like, at your job today, I ask people, viewing this, how easy is it for you to really write down every single exception that might happen in your job? I was a software engineer at Meta, and if people ask me for my job, I'd say, well, yeah, I code all day. I'll get A task and I'll work on it. But that's not the reality, right? The reality is I have meetings, you know, this happens, something breaks in product to go fix it. That's what we're getting at here. The second step is, oh, it's copied into a spreadsheet. Same thing here. You get the idea. One's a real 1, 2 are stale data validation, it's rekey by hand, columns drift. Same thing with checking into an internal system. I won't get into all this. You get the idea. Every step is extremely complicated. So that's what understanding the work really means. It takes time and it takes effort to sit with a person responsible. And sometimes multiple people responsible.
- [19:41]
A
Usually. Usually it's multiple people.
- [19:44]
B
Very, very often, yeah. I mean this company has like 5,000, 10,000 people working in it. Chances are you have a lot of people working on the same thing. Then you decide how the work should operate when intelligence is built in. So again, where does deterministic software live in? Where does the agent act? Where does the human approve? Where does the record get updated? I think the best combination of, sorry, the best solution of AI for most companies is a very good combination of deterministic software. Probably the majority of it is that, but then obviously the judgment that API calls to LLMs can provide. And then finally human in the loop. So this is just a fancy little digest, but it's really the agent that can then be deployed into existing systems. It's doing the first half, which is intake, validation, agent drafting. Then you have a human in the loop for approval. This is something that I strongly recommend my fdes to push for in an agent implementation. Once you hit approve, it'll then go through the latter half of the steps and that's what you're building. So the job of an FDE when you're building has three parts. It's obviously auditing, then creating evaluation suites to make sure the system behaves correctly. This is extremely important in the AI age. And then finally deployment, which is both hand holding the client to make sure they're adopting it, it's working well for them. But then also the software side, making sure that nothing breaks. You're monitoring all the metrics that matter, you're monitoring KPIs, SLAs, and everything needs to be top notch for somebody to really trust you as an fte. And every stage is a prerequisite for
- [21:09]
A
the next so for an eval. So prove the system behaves correctly. So in a scenario where the outcome is non deterministic, meaning it's tough to say what success looks like. How do you create an eval? Or can you create an eval for more creative tasks or tasks that are hard to understand if it's successful or not?
- [21:36]
B
Yeah, obviously for more non deterministic tasks, it's much harder. It's much easier to say, okay, was this email categorized correctly? Because we have 10,000 previous emails to go off of and it'll be the basis for our eval set. But even for tasks where it is non deterministic, like for example, creating a presentation, right, there's a million different ways to do it and sort of the beauty is in the eye of the beholder, where, you know, one thing that looks good to me might look bad to you here. It's very helpful to have as much previous data as possible. Obviously, if you have 5,000 previous presentations to go off of, it makes it a lot easier to create this golden data set of what we think matters. You know, you can say, you know, always put the logo in the top left, always have larger font of the styling, et cetera, et cetera, but this is obviously where you'll never get to a perfect result with just evals. You need human in the loop feedback to make sure that going forward, you at least have a feedback mechanism that improves your harness. If not like post train or fine tunes the model that you're working under. So on one hand, get as much data as you can and kind of determine what looks good, what looks bad, identify what matters to you, but also then always bake in human loop feedback, because even with a good data set, even with good evals, you'll need to have them constantly improved. And that's where that feedback mechanism comes into play.
- [22:57]
A
And I've noticed in this entire presentation, this entire podcast, we haven't really spoken about which LLM to use. Are you basically agnostic in terms of working with anthropic or OpenAI or Google, or if you're an FDE, basically, how do you think about which LLM to work with?
- [23:21]
B
Yeah, great question. Actually, something I probably should have touched on. We as a company are extremely model agnostic. So. So we think our value lies in our ability to be switching from one model to the next and making sure that your accuracy only improves, your cost only goes down, and you're not marrying to one intelligence provider, which we think will be an asset going forward. You don't want to monopolize your intelligence, your inference layer. That being said, if I was an FTE today, or if I was trying to become the best FTE today, I would stick to one model and one agent building platform. OpenAI has one, Claude has one agent SDK, et cetera. Every single model provider has one. Get very, very good at one of them because that will be the foundation that you then, you know, okay, let's, let's try out Claude tomorrow. If I'm already good at OpenAI's, you know, agent building platform, okay, I feel more confident about that. Let's go to Kimi3 or GLM 5.2. Let's see what the open source models can do. Let's build a proprietary harness. That's how I would go about it. But I wouldn't really worry about being model agnostic when you're starting out as an fd, because that's again, not where your value lies. Your value lies in how good are you at understanding both sides of the aisle? Because that can then apply to any model.
- [24:30]
A
Totally. And it's, we're getting to a point where the models are very similar in a lot of ways.
- [24:40]
B
Yeah.
- [24:41]
A
And a lot of the big players are, you know, they have like, Google will have their frontier model, but they'll also have an open source model, for example. And so now you're getting to this place where it's like, okay, you can play with their open source model, you can play with their, you know, frontier model. And so I expect that the arrow of progress around LLMs is they're going to have a bunch of different products for you to play with. So. Yeah, I agree. If you want to pick an ecosystem, bet on an ecosystem that you believe in for whatever reason, be the best at that ecosystem, and then as you become the best, then it's like, okay, if you want, and you're working with a client, for example, and for whatever reason another model makes more sense, then great, you know, you can go and recommend that.
- [25:34]
B
Absolutely. Yeah, I would say that that's exactly right. And then just to really hammer the last point in your ability to determine what model is best for a task relies on your understanding of various different models. You're benchmarking them along the way. But to your point, don't put the cart ahead of the horse, like really get good at one before you then try to venture out and make that understanding. I would totally agree.
- [25:56]
A
Yeah, I mean, that's like, yeah, you don't want to like, hammer a specific, you know, model when you don't understand what the system is a set of tasks are. It's like the equivalent of, you know, you're a waiter and you Just hand someone a glass of Pinot Noir and they're like, I didn't ask for that. You know, you, you know, a good restaurant has a sommelier, and sommelier's job is to understand what is your palate. Do you like dry wines? Do you like wines from, you know, France, you know, southern France or northern France or, you know. Yeah. And that's the biggest wine guy. So, you know, the analogy might break down, but that idea, I think of just like understanding what want first and then deploy makes a lot of sense.
- [26:52]
B
Yeah, I mean, honestly, I thought that was pretty good. Like, sommelier of agents is an fte. Like, you really go in to figure out what they want and then you give it to them. Right. You might give everybody Pinot Noir. It might work for some of them, but it's not going to work for most. And that's why, again, most AI pilots fail 100%.
- [27:08]
A
Cool. All right, let's keep going.
- [27:11]
B
Sweet. This is again more of the same. Find the workflow worth rebuilding. I'm going to link this. I'll have Greg link this document in the channel. So if you just want to dive in deeper in here. But the idea is again the same. Collect the context, trace the FD findings, figure out the bottlenecks, the repetitive work, the judgment points, all that stuff, and then produce the operating map. And this back and forth is why having the understanding of the business and the tech is super important.
- [27:37]
A
Cool. By the way, if you go back to the audit, if you want to be an fde, we just had an episode with Corey Gannim, who he came on the podcast and he basically talked about selling audits as a way to learn about someone's business and then deploy AI afterwards. You can sell the audit, right?
- [28:01]
B
Yeah.
- [28:03]
A
In charge for the audit and then the implementation. You can charge like a monthly fee or you can charge a one time fee. How should people think about that?
- [28:12]
B
Yeah. So actually we're in this exact business of implementing AI across the largest companies on the planet. And we require every single engagement to start with an audit, which, you know, obviously costs money to the business. This is extremely valuable. I think, again, like, there's a lot of misunderstanding. Like, you know, you can just throw AI on the, on the, on the, on the company. The audit is worth so much money to a business. I mean, we've had companies that tell us like, the audit was worth 10 times what they paid for it. So it's better than McKinsey because it's so telling. AI is so new. Like you said, no One really understands how to go about an audit. But if you're able to say like, look, here is in your department, here are all the different workflows and we've mapped them out very cleanly. Right. We have the full steps, the back and forth, the exception handling. We're going to map that out for you and we're also going to tell you what we think is worth automating versus what isn't. Give them that priority map, give them that matrix, that ROI matrix and then go ahead and show them how you would build it and show them the roi, show them the use case that is worth so much money to a company. I think this is something that even most consulting firms are not able to figure out. And this is where you have an edge. If you are really up to date on AI and you actually know live and breathe it yourself, the audit is worth a ton of money to a business.
- [29:35]
A
Totally. It's also a chance for you to like build trust with them and show them how you work and under promise and over deliver and it gets their creative juices flowing around like, okay, I didn't realize that because you're producing an operating map. Right. So you didn't realize like, oh, hey, I never thought about that use case or I didn't think that this would produce this expected business value. Maybe it's worth investing in.
- [30:09]
B
Absolutely. It's funny, you know, when we first started, you know, the company was last year. This is before FTEs are really a big thing. We used to call the audit the medicine that neither one of us wants to take. Right. Like a lot of companies are like, oh, like do I have to and audit. Can't you just like token max and start building? But it really is so valuable and they realize that as the audit goes on. So I totally agree.
- [30:32]
A
Yeah. Because we also have an agency called lca and LCA is well known for building, working with the biggest companies on the planet and then taking their products from a product perspective and bringing them into the AI age. So you work with a Dropbox and what is an AI first version of Dropbox or Slack? An AI first version of Slack look like. And we started doing audits as like, hey, let's audit your product first. What we noticed was the word audit was a tough pill for people to swallow. And we just rebranded Audit as a Sprint. So it'd be like it was a design sprint. We just kind of like. And we like, you know, brought in the concept of an audit with it. So we noticed that worked better. So just, just A little tip for folks.
- [31:31]
B
Super helpful. For some reason, people have an allergic reaction to the word audit.
- [31:36]
A
Well, I mean, they think of. They think of like a tax audit.
- [31:38]
B
Yeah, yeah, fair enough. The AI audit doesn't have the same ring to it for sure.
- [31:46]
A
Yeah, Cool.
- [31:47]
B
All right, let's keep going again. Deterministic software versus an agent versus a human in control. I'm not going to beat a dead horse. You got to prioritize the high volume workflows where the improvement is large enough to matter. That's your job as an fd, is to figure that out firsthand. Evals, you turn non determinism into evidence. You got to make sure that you have the right data, the required steps, it matches the expert and it's safe to act on. And you got to make this kind of matrix. And wherever you feel like it's not safe to act on, you route it to a human and you create an evaluation report. So you have 50 runs and 41 of them passed. Of the nine that didn't, let's investigate why. Five of them had missing data, four of them had the wrong record pulled, and then you use that to improve the system. Greg, you touched on this earlier. Like, how do you set up evals? This is sort of the framework that I would use.
- [32:37]
A
This is cool, by the way.
- [32:40]
B
Yeah, it helps to just have a decision tree, a matrix, when you're thinking about things again, because everything is so new. You could go a million different ways. But I'm sure there's other ways to do this. But this is ours and this is just how we kind of think about things at a high level. It depends on the case, like spaces, for sure. So how do you make it deploy and work inside the business? This is again, phase three. The first one is the audit. The second one is evals. Third is deployment one. We really preach about integrating with what already exists. I think a lot of AI folks are forcing migrations to new software. And the reality is, and this is a tip that I give to all my FTEs, you have an edge if you're able to build on top of their systems. So, you know, one of our clients, for example, said that they spent, you know, a couple of years, a couple million dollars moving to NetSuite, which is an ERP software. And if your AI solution is like, hey, we have to make you move off of NetSuite, they're going to tell you to get lost. But instead, if you're saying, which is what we do, build on top of NetSuite, make it much better and integrate that NetSuite with your Salesforce, with your SAP, with your Concur, Expensify gong. You know, every other piece of software workday that is a much more powerful system and that is where all the value lies. Then if you can test it in a controlled environment and really scale up from, you know, deployment to shadow mode to increasing autonomy to then being deployed in production, that is going to be your edge as well, where you're not forcing a massive shift. You're kind of walking them through that journey. And to our point earlier of why you meet them in person, it's because it's a lot easier to kind of guide them along that journey if you've met them face to face versus if you're just a guy behind a computer screen saying, hey, now we're going to flip a switch and AI is going to run your business. That's a much different, much more polarizing approach.
- [34:29]
A
I mean, it makes sense, right? You did the audit. And then if you're going to pitch to them, hey, you've been working with this software stack for the last 20 years. All of a sudden, go switch to this thing and it's going to cost you a bunch of money. And there's just like so many unknowns. Like, that is a tough pitch to sell. You know what I mean? And like, you want to, if you're pitching anything, you want to pitch something that feels like you're fishing with dynamite. So it's like, how can you, you know, how can you fish with dynamite? You just say, like, hey, you're, you have this system and the stack, you're, it's, it's worked for you. I'm going to make it better and it's going to help you all be more efficient. It's, it's going to help you reach customers faster. It's going to drive value for customers. It could increase revenue. Like, when you start saying things like that, it's like, okay, no brainer, no brainer, no brainer. Also, you have to keep in mind that you're pitching to people at a company and people at a company. I'll say the thing that people don't say, which is they don't want to get fired, right? Like, they want to get promoted, actually. So your job is to help them get promoted. How do you help them get promoted is probably not by moving from one erp to another erp that may be marginally better. You help them get promoted by driving value cost effectively. If you can drive value cost effectively, everyone's High fiving. Right. Because when performance reviews comes around, the employee the executive can point to. I worked on this project. Yes. I worked with an outside agency like LCA or Veric agents or individual FD freelancer. But as long as you help them do that, that's what's going to help them get promoted.
- [36:33]
B
Yeah, totally. And just again, like double down on that. They view you as a risk. Right. They can just sit by and let things stay the same and status quo, they'll be fine. But if they instead bring in an FDA who's gonna change stuff up and maybe it fails. Like they're worried if this fails, it's a terrible look on me. Forget like migrating to another erp. Even just you being involved at all is a risk to them. So you have to de risk this as much as possible for them if you really want to sell yourself into a company. And what I strongly recommend is like, do the audit for free. Like, get your foot in the door, prove value there, come up with a plan, and then only get paid when you really prove measurable value. That is what I strongly because that de risks the whole thing. Your first few customers, if you're really starting this out, will teach you so much. They are genuinely worth more to you than you are to them. But after you have 1, 2, 3 of those, then you can start charging for this because you're going to be leagues and miles ahead of everyone else in the space, I promise you. It's still so early. I know this from our company as well. There is so much demand for people who really know how to do this. And quite frankly, there aren't that many people who know how to do this. Get started, get your feet wet and really prove that you know what you're doing. And that's de risking the whole thing for them.
- [37:58]
A
Totally. And it's just going to give you the confidence too, you know? Yeah, right. Which is important.
- [38:04]
B
Yeah. And you'll know what matters to them when you're selling. Like, you can touch on different aspects that speak better to this person in the function. It's all really valuable. If I had to put one page cemented burned in everyone's brain, it's this one, which is you go from audit to evals to deployment. There's some steps in the way you build, you observe and you improve. And the loop runs again. Because once you improve one system, the next one becomes extremely clear. There's always interconnected bottlenecks where one workflow is impacted by something upstream and it frees up something downstream. And this is why AI is so pervasive in an organization. It's because once you have it in one place, you're going to need it everywhere else so that you're not just, you know, 10xing one workflow, 10xing another. You're 100xing the entire business as a whole. And that's your job as an fte, is to go from audit, to evals, to deployment, over and over again. And the next stage that I'm going to show is the 30 day plan of how you can get there from zero to, to one. If, if I was starting from scratch, how would I go about it?
- [39:10]
A
And you did this, by the way, you started from scratch, right? I didn't know this, but you were, you were an engineer at Meta, right? And then you sort of learned how to do this. So you're speaking from experience.
- [39:21]
B
Yeah, absolutely. I was an engineer at Meta for a few years, working on a different couple, different products. But I was never a consultant. I would never really understood what mattered to businesses as deeply as I do now. And we got started again just by by doing it. And we had this thesis that AI needs to be applied and it allows us to get ahead of the curve. But you never learn by reading and you only learn by doing it. So the goal from this is to do. If I could condense what I did over a year and really had the biggest learnings, the biggest WINS in just 30 days, this is what I would do. So the first step is build an agent that can complete a real loop, right? Build an agent that's actually useful as a workflow. So like Ask ChatGPT, what is one real enterprise workflow in a function of the back office, right? It could be finance, it could be hr, it could be procurement, logistics, it could even be front up, could be sales, anything. Get the workflow in as granular of a detail as possible and build an agent for it. Even today it's very hard to build agents, right? We think that it's a solved science. It's not. There's a thousand different ways to do this. Everyone has different definitions of an agent. My definition is this. If I give you a task, can you solve it in as much detail and as high enough accuracy as possible? It's different than me prompting Claude to go do it. It's far more in the background and it has much more of a repetitive motion where I'm not reliant on somebody prompting perfectly to make it happen. I can prompt like an idiot and it will still happen. That's my kind of requirement for you when you're solving for this. There's a bunch of different, you know, aspects to this, but if you have seven days to work on this, you'll be able to pick it up. Agent looping, then tool usage, then guardrails, then context and memory, then the audit trail, which is incredibly important. I want to hammer down here a little bit. If you can't show the client what the agent is doing, they will never trust you. There's a big fear in AI agents today that okay, it's going to go off and do something, it's horrible. There's a lot of fear mongering as well. I won't say from who, but everyone knows who I'm talking about. You need to show that the agent traces are logged, everything. And this is a software engineering problem. So if you can do that, you're a step ahead. And again, there's a full day for each of these things. To clarify, if you're, you know, working 12 hours a day, I don't expect you to be able to pick all this stuff up perfectly on each every single day. But this is what I mean by like a 30 day plan. You can space it out as you need. It's not like you have to get it done in 30 days. Then a real workflow, then a checkpoint. The checkpoint the last day is you have a working agent with tools, guardrails, deliberate memory and a full audit trail for one task. You might not even understand the task the best, but this is just to get you well versed in building agents fair. The second week is turning that demo into a system that can recover again very heavily on the engineering side. So defined JSON schema, not freeform text, you're validating schema, you have failure modes, and again I want to call it failure modes, exception handling. And we also see on the 13th day as failure handling is also extremely, extremely important. This is where going in deep into a client matters because if you understand, hey, when something goes wrong, how does it go wrong? And let me build the agent around that, that is extremely important. It's far more effective than you're building an agent that solves for just the happy path. It's called the happy path. If you're building for the unhappy paths, the 1500 different ways it can go wrong, your agent is worth a million times more. The way I say it is this. There's only one way that something can go right, but there's a thousand different ways something can go wrong. So if you're only Building for the way it goes. Right. You're worth nothing if you're solving for all the exceptions. That's where you are worth something as an agent. That's week two. Then week three is where you start to make it measurable and economically viable. Right. So you'll have the retry logic. Yes, this is more engineering. You have the golden data set for evals. You'll make sure that it improves over time, but you'll also start understanding, okay, this is what we talked about earlier, Greg. Looking at cheaper models for subtasks, looking at less soda, less frontier models. Can we get this job done with a Gemini Flash, or can we get the job done with a Muse Spark? Or probably not a llama 4, but there's other models that can be a good fit there. This is where you start to do more of this test, which is we have an agent now. Now let's try to optimize it. Let's try to measure, okay, how much is it really moving the needle? If I deploy this in production, how much time am I saving? How much risk am I mitigating? How much revenue uplift do I have? There's only three buckets of measurement that matter for a business. It's those three. Revenue uplift, risk mitigation, and cost savings. So you need to measure your agent across all three of those buckets. And at this checkpoint, you have an evaluated agent with known failure modes, measured costs, and a golden data set. And the final week is defend the system like an fde, which is all of the business around it. It's the pain points. It's why AI belongs. It's the architecture behind it. It's the iterations. You know, when you first built the agent, it got this wrong, but then it improved over time. Accuracy went from, you know, 70% to 95% and the economics around it. So how much time did you save, the error reduce, again, risk, revenue costs. We talked about this, and you rehearse this as an engineer. So what was the architecture? Were the decisions that you made? And then you also rehearsed this as a vp. Like, what was the problem that you solved? What was the outcome? What was the evidence? What was the risk? And this week is where you're going to know. You know, was the system that you built worth the salt? Was it worth the investment? How much could you charge for this when you build it for a customer? That's what this week is for. And I strongly recommend that during this week, you pitch your agent to businesses because they will tell you, like you know, did I get. You'll, you'll pitch them, did I get this right? Did I get the economics right? Am I thinking about this the right way? And they'll tell you point blank no or I want it built in this way. And you'll start to see like okay, now for my next FTE engagement. Starting out with an audit when you're actually embedded with a customer, you'll learn much more about that. Obviously this 30 days is you're not embedded with a customer because you can't. Because you have to become an FDE first. That's when you can finally start to pitch yourself and be involved in a company. So if I had to zoom out, this would be the 30 days it's doing the job before you have the title. So on day 30 you understand for deployment engineering, but you also have evidence that you can do it. And if you pitch this to a company, they'll be much more likely to give you a shot. And that's my goal.
- [46:15]
A
Foss, this is perfect. This is exactly what I would recommend too. What would be so cool is if you actually taught people how to do this right and spent 30 days with people to actually do this. Maybe we do it together. Just an idea, if people are interested, I'll just include a link in the pinned comment on YouTube. I'm just curious if people people are interested in like. Because it might feel overwhelming for people to do this on their own. I mean I still think you could do it on your own by the way, but I wonder, and by the way, I'm not promising anything, I'm just curious, are people into, into some sort of program for this? Because they don't teach you this at school.
- [47:11]
B
They don't. They should. I'm sure we'll have university courses on FTE soon, but yeah, people are 24
- [47:17]
A
months, it just takes so long. I remember I was in computer science school in 2008, no, 2009 and at university. And I remember the App Store had just come out. It was so clear in 2009 that mobile apps was the next wave. Just like it's so clear right now that AI agents and AI is not even the next wave is the wave. And I just remember the textbooks at the time and I went to a top university textbooks at the time. The course material was like building old school software. And I remember going to a teacher, a professor, well known guy and saying why can't you teach us how to build in objective C and to build for the app Store? And he was just like, yeah, it's just not in the textbook. Just not in the textbook. And that's when I like, I was like, I'm gonna drop out of this. I'm dropping out because like, I don't want to learn yesterday's stuff, I want to learn tomorrow's stuff. Now there's always the argument to be made that you need foundational work. And so like I learned a lot in university around like foundational stuff, around maths and physics and stuff like that. And I actually think that that stuff was really helpful just in like learning how to think. But like the actual tactical stuff did not really learn.
- [48:53]
B
Yeah, I, I do. You know, I want to say like this time is different, like just because it's so powerful. Like AI is so like you said, it's the wave that I'm hopeful that universities are going to pick up sooner than later. But for some reason I feel like you're right. I don't think it's going to happen anytime soon.
- [49:10]
A
Yeah. Well, there you have it folks. What? What? FDR how to become one a 30 day plan. Vas, anything else you want to share?
- [49:29]
B
I think like you said, you might not find this in university, but you're absolutely going to find it on YouTube. Greg is teaching everything that you need to know and Twitter as well. Those two sources are going to be where everything is released. I mean even Mark Zuckerberg had to come back to Twitter who announced the latest model for meta. That's where everything's happening. Study the game there and you've got a great coach right in front of you with Greg. So hopefully that, you know, people are really taking advantage of this time where there's a significant alpha from going out, learning, doing it yourself, being scrappy with it versus waiting for a university to come by and teach you this because that's not going to happen anytime soon.
- [50:12]
A
And it's free, right? You can listen to this and apply. It's free. So it's just like why not? Right? Yeah. Voss, thank you for being generous with your, with your as we see on the channel, the sauce and the tactics and just like breaking this down. So clearly I've been following you for a couple years now almost and you're a must follow. I'll include links on where you can follow Vaas from Varrick agents in the show, notes in the description. You know, please comment what you what you thought of this episode. Because I enjoyed myself with Vaas. I'd like to have him back on the podcast again. Hopefully he's down to come back on but please let us know. I read every single comment you want to like and subscribe for more of this in your feed. Voss, any last words for the people?
- [51:02]
B
Greg, you're a legend. Thank you for having me on. To the people, I believe in you. I really believe this is a fundamental shift in how work is done and you are. If you're listening to Greg and you're on this pod, you're watching this, you're already a step ahead. I'll be reading every single comment too. If any questions you have for me, let me know. But I would say go out and get it. Go out and get the job done. Make the most of your ability to understand AI. And it's still so early, so get ahead of it while you can. Greg, thank you for having me on, man. You're a legend.
- [51:31]
A
Amen. All right, catch you next time.
- [51:33]
B
Cheers.