Loading summary
Transcript52 lines
- [00:00]
A
Foreign. Welcome to the AI to roi, the Big Story edition of the podcast. I'm Ray Reich, founder and CEO of BenchmarkIT, and I'm joined by my Big Story co host, Peter Buchanan.
- [00:21]
B
Yep, I'm Peter Buchanan. I am the founder of New Plane.
- [00:25]
A
So, Peter, you know, this week we published a pretty interesting edition of the newsletter all about AI in budget. And to me, this is something that every CEO and CFO should be thinking about majority. And probably already are. But I think it's a financial do or die. And that is, where is the future AI budget going to be coming from? How do we track the cost? How do we, how do we track utilization? How do we track business impact? And who is responsible for managing all that?
- [01:03]
B
Wow. Well, the short answer is for most enterprises right now is that nobody's really tracking it very well. And so we're going to walk through why that's a problem, where the evidence shows up in earnings reports and financial statements and what the path forward looks like. So we're going to follow the structure of this week's newsletter, which is the AI to ROI Big Story. The AI budget dance is underway, which Ray and I co authored. We have five sections to our podcast today, each one building on the last. So let's get started. So the headline in the newsletter is that token prices are down 98%, but enterprise AI bills are up 320%. Ray. So how does that actually work?
- [01:51]
A
You know, Peter, you know, because what I have done for a living for the last almost six years is benchmarking. And a couple years ago when you and I first started talking about AI to roi, I really had this hypothesis that AI costs are going to grow rapidly and organizations, companies aren't ready ready for it. And one of the common things I was told, and I read on LinkedIn is don't worry about it. Just like every other technology revolution, the per unit token costs are going to fall, which they are. But it's, in my opinion, it's masking a real explosion in total AI spend. And that explosion is because a lot of companies and a lot of departments are experimenting. Utilization is going way up. We think about AI coding assistance and, you know, now I think like 84% of developers in the US are using them and we're just beginning this AI deployment kind of generation. So usage is going up much faster than the costs are going down. But let me focus on cost for just a minute. Like if you look at GPT4, which by the way, seems so old now.
- [03:11]
B
It does. It's a Year old, but.
- [03:13]
A
Yeah, but you know, the cost in late 22, early 23, was about $20 per million tokens, and it's down to on average 40 cents per million tokens. So that's a 98 price reduction. And the Next Web published an article just a couple days ago that the average enterprise AI budget has grown from 1.2 million in 2024 to 7 million per year in 2026. So it's growing dramatically. And in that same report, it talked about how a single chatbot prompt. Hey, Claude, tell me how to write this email. You know, in early 23, that cost about 0.4, which was pretty good. But now we're doing these holistic systemic end to end workflows, and we're just starting that journey. And in 2026, per the next Web, the average cost of that is $1.20. So 30 times more than what a chat prompt cost three years ago.
- [04:19]
B
Right. So there's this Goldman Sachs report decoding the agentic economy, and they project that AI will dive a 24x increase in global token consumption by the end of the decade. So that's 120,000,000,000 tokens processed per month, which sounds like a lot of tokens. Enterprise workflows, not consumer chatbots or even just you and I asking questions of Claude. That's about 70% of the projected volume by 2040. These numbers describe this absolutely gobsmacking trajectory. So what does it look like when the trajectory meets an operating budget? We've both been CEOs before. You are still the CEO. And this would keep me up at night. I can tell you There are three situations in the end of 25, beginning of 26 that show what happens when CEOs and CFOs don't pay attention. So let's go through those because they're pretty shocking.
- [05:32]
A
Hey, I just had to comment that 120,000,000,000 tokens process monthly. I'm like, so when does Elon Musk become a quadrillionaire?
- [05:43]
B
Yeah, exactly, exactly.
- [05:44]
A
That's another story. Okay, so three big stories have come out in the last couple weeks about token consumption. So first it was Uber. So uber gave their 5,000 engineers access to CLAUDE code. They incented usage of that by tracking token usage on internal leaderboards. And they believe that treating consumption as a proxy for commitment to becoming more productive. Peter. So that was done in late December.
- [06:16]
B
That didn't work.
- [06:18]
A
Early April, they came out and announced we we've just consumed our entire annual token budget for CLAUDE code in less than four months. And then Uber COO came out and said token consumption showed no measurable correlation with useful consumer facing product features. Pretty damning, right?
- [06:42]
B
It is pretty damning. But you know, there's another episode we could be had about the. That you could have about the development pipeline around coding agents and how things get messed up further down the line when you use them. But there's another one here from a very famous company.
- [06:58]
A
Yeah, and it was Microsoft and their cfo. Right. They revoked the use of cloud code licenses in May of 2026 for their engineers because their engineers were spending anywhere between 500 and 2,000amonth. So maybe not blowing the entire annual budget in three months, but an issue. And then the big story, which came out just. Ooh, big story.
- [07:22]
B
That's.
- [07:22]
A
I wasn't even meaning to say that, Peter. But the big story was an unnamed enterprise, which may or may not be in Washington state, ran up a $500 million Claude bill in. Wait, wait, a single month. To me, that says they must have had no usage caps, no cost visibility, and little to no governance in front. Because even for the company with a goal to maximize AI coding tool utilization and competency, I can't believe that they had planned that this could go up to $500 million a month.
- [08:01]
B
Right. And they got to hope that name of that company never leaks. So far it hasn't. So. But on the other side, the major AI vendors are absolutely making bank, at least for now. We've talked about before. Anthropic's annualized run rate at the end of last year was $9 billion. It was $47 billion as of mid May. More than 1,000 enterprises spend a million dollars a year on Claude. Eight of the ten Fortune 10 customers use Claude. They've got their S1 filed looking at an IPO in the fall. OpenAI is basically right behind them. Running at about $33 billion in annualized revenue as of May. Its API now processes 15 billion tokens per minute. That's a lot of tokens up from 300 million tokens three years ago. And that's a 50x increase in three years. But. So let's dive deeper into the Uber story. What's the lesson here?
- [09:11]
A
Well, I think part of it is this is more of a governance failure than a technology failure. And quite frankly, it's a motivation issue. Right. And we've heard about token maxing. We've talked about it before. Leaderboards which measure and incent token consumption as a proxy for AI commitment or even competency. That's a mistake. And the CEO's disclosure that we're not really getting output as measured by accelerating new features that the consumers are using, that's key. And now they came out and said reporting says they now have a fifteen hundred dollar per engineer per month cap. Now that's a control, right? That's a governance. But until I hear how they're measuring the business value and impact, is that accelerating velocity of product releases, is that driving more customer engagement, customer satisfaction, Right. I'm not. I don't think they have a real good strategy in place.
- [10:20]
B
They haven't showed anything public, they haven't said anything public about what, what they've learned and done other than saying we're going to cap our usage per month.
- [10:29]
A
Now Microsoft and their revocation of thousands of cloud code licenses right at the end of the fiscal year, to me that suggests it wasn't a product or even business impact grounded decision, it was a budget and cost control decision. So now if she could link workforce reduction or increase productivity that would be one thing. But all we heard in the news was they were eliminating cloud code. And then I think about the other side of this double edged sword and I look at the revenue trajectories of Anthropic and OpenAI and by the way I got to highlight this when they report their annual recurring revenue, it's not true ARR its recurring revenue run rate which is take last month and multiply by 12. And I think so much of that is happening from non governed, non well managed experimentations like the Uber, Microsoft and the unnamed company we talked about that, that revenue trajectory, the growth trajectory I do not think is going to be maintain but I may be going off topic there.
- [11:44]
B
No, I think you're on target. No caps, no attrition, no governance. That's, that's not a rogue event. It's, it's deploy. It's the default posture for a lot of companies right now. But I, I bet it changes quickly. So, so the AI spending is real and it's accelerating. So what is the full scale of what it looks like over the next few years? So let's just do tee that up a little bit. Software and token spend is transitioning from a footnote in the IT budget to a category that will rival or exceed total IT expenditures within a decade, which is amazing. So the math should really change how every CFO talks about where, where dollars are allocated in their OpEx. So Ray, let's talk about the arc of AI spending because now these are going to be macro numbers and Then we're going to dive down deeper after that.
- [12:46]
A
Okay. Hey, for those listeners, if you're getting off your elliptical or getting out of your car, pause. Yes, this is a section you're going to want to really listen to, and there's a lot of data here. So, number one, AI software and token spending. It was roughly $100 billion in 2024, but maybe more importantly, that was less than 2% of global IT spend, per a report, and I believe it was Gartner. AI software and token spend is on track to reach $2 trillion by 2030. Now, that's a 20x increase in less than four years, and that will be more than 20% of all global IT spending. And Gartner reported that agency AI software spending is rising 141% in a year to nearly $202 billion this year. And for context, the entire SaaS marketplace, which has been growing for 25 years, is about 250 billion. So they're going to be close by the end of next year. Peter.
- [13:59]
B
Wow. Well, the SaaS comparison is very interesting because AI token and software spend already approaching half the size of the SaaS market. SaaS taking 25 years to get there. The category is it's just a candidate. Money's got to go from one place to another. It seems like it's a candidate right from some of the places where AI is going to be paid for.
- [14:27]
A
Gartner's not long alone. Oxford Economics, they projected global AI spending will become about 23% of total enterprise technology spend by 25, up from, they said, 4% today versus 2% from Gartner. But regardless, what this says is the AI cost and token spend is going to increase dramatically. And where is that budget going to come from?
- [14:55]
B
Right. That's part I want to get the key question, where does that budget come from? Because there are only so many places. There are only really, I think, two budget pools that are large enough to absorb AI spending. So the software budget, which we just talked about, because a lot of SAS software is turning into AI software, and the labor budget, because it's the only category large enough to fund AI at scale. So the evidence shows already that both of those budgets are being tapped. So let's go into that, Ray, because these numbers are astounding and they're just getting started.
- [15:32]
A
Yeah. And I don't know if I buy this, Peter, that there's only two budget pools. I think, because if you look at software spend or SaaS spend right at 250 billion, maybe growing 8 to 11% a year. That's not enough to fund the growth we just talked about in an AI and token spend.
- [15:52]
B
Yeah, absolutely. It's just a down payment.
- [15:55]
A
Yeah. It could be the overall IT spend. Right. People are estimating AI would be 20 to 25% of that. But it can't all come from your SaaS budget. So what are you going to be giving up in it? Your security? Nope, you need more of that with AI, your networking infrastructure. Oh, I need more bandwidth with AI, my server and hardware. Oh, I may be having to deploy my own GPUs and TPUs, so it's not going to come from there. So this is something that a lot of people don't like to talk about. CEOs and CFOs definitely don't want to talk a lot about this due to press, but today's newsletter and this podcast. Here's what I think. I think labor, because it's the number one source of operating expenses today for any type company. And healthcare labor's 41 to 45%. Professional services company is up to 50%. Technology and SaaS labor is 20 to 35% of total revenue. Manufacturing is a little bit lower, 12 to 20% because we have more capital equipment, et cetera. And retail it's 8 to 15%. Insurance is 9%. But as you can see, it's a much larger percent of a company's entire budget. And as we said, software and IT, even though it's a pretty big budget, it's only 3 to 4% of revenue. So if I'm looking at a bucket that has cash in it, that's 3 to 4% of revenue. And another bucket that has 25 to 40% of revenue called labor. I might be looking at getting some of that cost out of the bigger bucket, the labor.
- [17:45]
B
Right. AI is supposed to streamline how labor is used anyway. So there's a scenario down here from Xylo ray on their SaaS management index. Let's go through this scenario on their data on software licenses and where at the beginning companies might find some money.
- [18:11]
A
Well, Zylo, which by the way, I had the CEO on the predecessor to AI to ROI. The metrics of major up podcasts. Great podcasts, but they have over 40 million software alliances under management. That totals 75 billion of spend under management. And their data shows that AI native application spend was up one hundred and eight percent year over year. That's software spend and up almost four hundred percent in enterprises with more than ten thousand employees. So we're seeing AI Native software spend go up. Right. And 61% of the IT leaders that Zylo spoke to and surveyed, they said that they were forced to cut projects due to unplanned increases in AI cost. And by the way, that's the same exact research we found when we did the state of AI pricing through a buyer lens, like I think it was. Almost 80% of companies had exceeded their AI budgets for 2026 on a run rate basis.
- [19:15]
B
Right. They have to be able to pay for it. So SAS is a candidate. So then there's the other evidence that there is evidence out there that AI is actually affecting spending in the labor market. Right. So. So according to Challenger, Gray and Christmas, leading HR consultancy, as of mid May 2026, more than 113,000 tech workers have been laid off across 179 companies. That's 33% faster than in 2025. And 48% of those cuts are explicitly attributed to AI by the companies that are making them. And that's actually the number one. That's the number one category. A reason for these layoffs is I have AI, I don't need the person. So the first lever that enterprises are using to get this money is attrition. So if you look at a 5,500 person company with a fully loaded cost of 150,000 per employee, you got a 10% attrition rate. And that frees up $75 million a year in labor costs if you don't backfill those open rolls. So where does the money go? Well, an awful lot of it goes to the bottom line. They would hope, but probably an even larger percentage of it may go to actual AI applications and infrastructure. In essence, tokens, Peter, that's the key.
- [20:46]
A
I mean, you talked about the big layoffs. We've seen Oracle make some announcements, Facebook make some announcements, even Amazon made some announcements. Those aren't newsworthy. But what's not capturing the news in the press is the real way companies are funding AI spend today. And that's through not backfilling the attrition like you just said, hey, if you've got a 10% annual attrition rate for a 5,000 person company, that frees up 75 million. So at the end of the day, Peter, I have this hypothesis and I've got some instrumentation in place now to measure that, that what we're going to see as a proxy for where AI budget's coming from is when you start seeing revenue per FTE going up dramatically, because that's going to happen. And by the way, we haven't seen that dramatically in most industries to date and I've been looking at the last 12 quarters. But in my annual both private and public SaaS company benchmarks, we're seeing ARR FTE up about 25 to 35% across the board.
- [22:02]
B
Really? Wow.
- [22:04]
A
I mean if you look at public SaaS company spend, I'm going to do this off the top of my head. I'm sorry, not spend, but revenue per employee. You know, it used to be around $300,000 per FTE. Right now it's sitting at 400,000, a 33% increase. Peter.
- [22:21]
B
Wow.
- [22:22]
A
So I think this is what's going to happen and it's going to fly under the radar for a while. But it is a deliberate decision not to backfill attritive employees right now.
- [22:36]
B
Right. So we've got the great budget transfer happening, whether or not enterprises are actually managing it deliberately. Which brings us to the question of measurement. So how do you build the infrastructure to actually govern this?
- [22:55]
A
Well, it's going to be hard to build because things are moving so fast, but that's why it's great to see so many new software platforms out there that are purpose built to accomplish this. And I recently had the CEO and co founder of Laradin on a call and I also have spoken to a lot of other platform companies like Botano and their founder Alina and they are looking at. First of all, CFOs and CEOs are looking to gain visibility into cost. Just cost. Like that Uber story boy. Wouldn't have been nice to know halfway through the month that they were blowing through the budget. And then they could be more proactive in measuring it. So we got cost visibility. Then we're going to also need utilization visibility. Some other and these are more ad hoc conversations. People are buying Claude code, but it's like 20 to 30% that are really using it the most. Another 30 to 50% aren't even using it. So why in the heck am I playing for a cloud code license? Why? Because I don't have visibility into utilization. So we get cost, visibility, utilization. Then it's proficiency in productivity. How much productivity and how proficient are my employees? Indoor processes, leveraging AI and agentic AI. So you need to have visibility into that and then ultimately we're going to get to the business value, the economic impact, but you need to really instrument the entire cost, utilization, proficiency and ultimately the economic impact. And that's not being done right now.
- [24:37]
B
Well, some of it will be hard, some of it Will be relatively easy to do because you can calculate token spend as a percentage of revenue or you can calculate token spend as a percentage of employee. If you're a AI software company, you can, you can calculate token cost as a percentage of cogs specifically for software. So but the other things you're describing, you have to take apart processes and workflows to figure out how and why things are being used. For example, the CEO of Uber says, oh well, we didn't get anything out of this at all. Well, maybe they produced too much code for the downstream People in DevOps and QA and staging and all these other areas to actually accommodate it. Or they created too many and they produced so much code, maybe they produced a lot of bugs and so there's a lot of code sent back for rework. So those are the sorts of things that in order to get to the right number, I think people need to, companies need to really look at. Right.
- [25:45]
A
Hey Peter, this reminds me of when I was first getting trained at corporate America. One of the big topics was business process reengineering and let's apply it to a manufacturing company. Companies needed to understand their manufacturing process from a resource, time and cost perspective. Current state to justify multimillion dollar capital investments to automate and put robots on the line. Right. It's going to be the same thing for agentic AI. Hey, if we're going to increase token costs to that 3 to 8% of revenue that some research firms are predicting, we're going to have to truly understand current state, end to end process cost, future state process design and cost. And hopefully we're going to see that some of the cost takeout which are going to be people and some economic impact. That's going to be how we're justifying it. Companies are going to have to go beyond just token spend as a percent of revenue or token spend per employee or token spend as a percentage of cogs. They're going to need to get into AI and token cost per process and sub process units.
- [27:05]
B
Oh absolutely. The companies doing this measurement work right now, they're really building because they're not very many of them. So the ones that are actually doing it. So you and I have profiled a lot of reports and we've, we've highlighted them in the newsletter. There's less than 30% of companies that are doing really well with AI right now and are getting where they think they should be. But they're really building a competitive advantage by starting this early versus lagging companies and companies that are lagging they're going to have difficulty catching up. Ray, let's bring this to a close. What's the actual question, the actual question that every CEO and CFO needs to be able to answer regarding AI budgets and what is answering it? Well, actually look like, are we able
- [28:02]
A
to have visibility and measure the cost of our AI investments in near real time and are we able to manage it by understanding the economic impact of the input variable token spend to the output variable economic value? That's it. Now that's going to take significant time to lay in that infrastructure. But let's at least start with getting a better understanding of our costs as they're happening, what's driving those costs and what's our utilization trends? Let's start there. Now, the next question is and people already asking it in the media and boardrooms and I honestly think it's a little premature to have the answer, but it's not premature to ask the question what is the economic value or impact that will show up on our income statement that we can justify scaling our AI investment and then right after that or simultaneously, where is that budget going to come from? And if it's not going to come from short term revenue impact, which a lot of cases it won't, unless you're a software company, you're going to need to understand what other expense buckets can I reallocate those investments to this AI investment?
- [29:40]
B
Right. It's a big challenge. And also for CFOs, look how you handle attrition. That's probably the number one tool you have right now. Not backfilling is a powerful and quiet way to fund AI investment, but it only works if you're deliberately investing those savings which, which, and sometimes it's not immediately clear when it's working. So at least knowing that you're making a particular investment, it's a cost that you need to track and you need to do it basically as often as possible. So, so.
- [30:13]
A
Right, Peter, I'm going to wrap up just with kind of, yeah, I'm going to now say five metrics.
- [30:19]
B
Okay.
- [30:19]
A
The early ones are going to be easy. I want to know what my total AI token spend is as a percent of revenue.
- [30:26]
B
Right?
- [30:26]
A
I want to know what my total AI token spend is per employee. I want to know what my AI token, some people call it inference spend, is as a percent of OP X. And then I want to know if I'm delivering a AI enabled product, what my token or inference cost is as a percent of cost of goods sold. But at the end of the day the number that CFOs and CEOs should start tracking and look at the correlation to their AI investments is revenue per fte. That's a labor that's ultimately it's going to come down to is AI impacting my productivity? And the next I would go granular on the operating expenses and I would look at what my labor cost is fully loaded today, what my agent cost as a percentage of OPEX is today, and watch that trend line over time. Because my hypothesis is the companies that are effectively deploying AI will see their labor costs as a percent of OpEx go down, agent costs as a percent of OpEx will go up and revenue per FTE will, will go up faster.
- [31:43]
B
It'll go up faster. So the other thing that you didn't mention that I think would be very interesting is if develop a methodology, do you take process workflow and run it all through to say see where all the people and AI expenses are to figure out how you get value from that work, how you get value from that workflow. So you have to start that right now. At the same time, things that workflows that are very important or maybe there's an easy one to figure out to begin with that gives you some confidence but, but beginning to do it from a process perspective to get the answer to figure out where you get ROI off that process, I think that's a big deal.
- [32:25]
A
And to the listening audience out there, if your company or you know of a company that has great actual use cases of how they're measuring the impact of their AI, how they're getting budget for material increase in AI budgets, hey, reach out to me at Ray Reich on LinkedIn and I'd love to have them as a guest on the AI ROI podcast. And if you like this topic and you like kind of some of the insights Peter and I are sharing on our podcast here, go to the AI to ROI, that's ai2roi.substack.com and read the Tuesday, June 9th edition of the AI Big Story. Thanks everyone. And thanks Peter.
- [33:12]
B
Oh, thanks. Absolutely. We'll see you next week.