← AI to ROI
Loading summary
Transcript50 lines
- [00:00]
A
Foreign. Welcome Back to the AI to ROI, the Big Story Edition podcast. I'm Ray Reich, founder CEO of BenchMarket, and joining me as always, is my co host, Peter Buchanan.
- [00:19]
B
Yes, I'm Peter Buchanan. I'm the founder of New Plan. And Ray, it's great to be here. This week we're going to talk about a topic that, that really captures the moment we're living through. And that topic is with all the anxiety and angst and all the massive investment, there's a big question. And that big question is, is AI actually going to pay off?
- [00:44]
A
That's exactly right. Comes from our Monday, March 9 Big story. Will angst, agony and adversity of AI be worth it? And I'm really excited because the spoiler alert though is it will be worth it. It's just going to be how difficult is the journey going to be and how long is it going to take? So let's break down both sides of this story, Peter.
- [01:10]
B
You bet.
- [01:12]
A
One of my questions is let's start with the reality. The infrastructure spending is staggering. Can you put some more numbers behind that for the audience?
- [01:21]
B
I can. So let's start with the number that makes everyone's jaw drop. So just for this year, the five largest hyperscalers, that's Amazon, Microsoft, Alphabet, Meta, and that newcomer, Oracle, they are on track to spend over $600 billion on CapEx to support the delivery of AI. So 75% of that 600 billion plus billion dollars goes directly to AI infrastructure. That's GPUs, servers, data centers, H vac, electricity, labor. So to put that in perspective, Ray, that level of spending nearly matches the combined scale of the Apollo program, the interstate highway system and the national build out of electricity all at once in a single year.
- [02:16]
A
Well, tells me that things are getting a hell of a lot more expensive, Peter. Or this is just that much bigger. And by the way, when we talk about bigger, I think that's where it gets really uncomfortable. Because if you look at, you know, 23, 24, 25, the major hyperscalers, those companies you talked about, they funded, the majority of that build out entirely from their own cash flows. And that error is over because the aggregate capex for 2026 actually is exceeding their combined free cash flow. So they're tapping into the debt market. Like an example, Oracle spending 75% of its annual revenue on CapEx.
- [02:56]
B
57. Ray, don't, don't give them 57. 57, not 75.
- [03:01]
A
And Microsoft's at 45. Yeah, yeah, but those are ratios more typical of heavy industrial enterprises than software companies. Software has never been capex intensive. So is this partially what's behind all the talk we hear about credit risk for our $3 trillion private credit economy in the United States alone?
- [03:23]
B
It is. And the bond markets are really paying attention. So Oracle's five year credit default swap spread has more than tripled since September of 2025. So that's six months. Investors are genuinely concerned about their concentrated bet on OpenAI as an anchor customer.
- [03:45]
A
Peter, we've talked about this multiple times in the newsletter and that's the circular nature. In fact, OpenAI just raised $110 billion last week. And that came from people like Amazon who then committed, OpenAI then committed to actually purchase compute capacity from them. So very circular. Right?
- [04:09]
B
Amazon, Nvidia, both of them. And those investments were actually, it was almost more like a supply chain deal than raising $110 billion and their milestones attached to them. So the 110 billion isn't going in the bank immediately.
- [04:26]
A
You know, there's, there is a little bit of some gray clouds out there and one of them is that data center construction just can't keep up. So even if you have all this capital, supply is not unlimited. You can't just snap your fingers and build out an example. The world currently has around 12,000 data centers in operation with 3,000 more being planned. And by 2030 that infrastructure spending is on track to reach 1 trillion. But here is the. I think the silver lining and this is going to be a thread I'm going to pull through the rest of today's conversation. I don't think we're going to see AI build out happening nearly as quickly. So it's not going to be impacting jobs nearly as quickly because almost every one of those data center construction projects is behind schedule. And I don't see it speeding up in the next 12, 24, 36 months. What do you think?
- [05:23]
B
Well, that's absolutely true. And power is absolutely the number one obstacle. So grid upgrade timelines in the United States are basically eight plus years, which is incompatible with the two to three years it takes to build a data center. Plus everything else the data center needs is an incredibly short supply. So you know, AI chips, memory servers, networking gear, storage, transformers and H vac skilled labor, even construction materials. And most those shortages aren't supposed to really sort of be solved till a minimum till 2028. So that's two years from now.
- [06:07]
A
Yeah. You know, this reminds me of a saying, and I believe it's called, is it Amherst Law, which Is, you know, we drastically overestimate the impact of technology in the short term, but we underestimate the impact in the long term. And I think we're overestimating the impact in this short term because of all these supply constraints. Because we got this massive demand for compute, we've got the capital to build it, at least we're spending the capital to build it, but the physical infrastructure just can't scale fast enough. But once again, that thread I wanted to pull through, I think there's some positive behind that supply constraint because now companies, humans, the employees who current jobs are at risk due to agentic AI. It's going to give us a little bit more time to create those new AI era economy jobs to actually make up for kind of the lost jobs we're going to see in a short term. And I also think it's going to give corporations the ability to get better, better at these agentic AI and AI program deployments because we still hear all the time, we're just not seeing enough production level roi. So can you talk a little bit about that, Peter?
- [07:25]
B
Sure. So everybody talks about the pilot to production problem. And so let's talk about what's really happening inside enterprises because most CEOs know that they're not getting what they thought they were going to get out of AI in the short term, and that the process of getting something into production is hard. A couple weeks ago, you and I did a podcast on this Deloitte survey that came out at the end of the year. There were almost 2,000 senior executives that they interviewed globally, in a couple dozen countries. A typical technology implementation of a software product, it takes seven to 12 months to get payback. Only 6% of AI products are delivering returns in under a year. But I also think that the expectations of those returns are much higher than the typical software project or project that's being delivered. How do you think about the gap of expectations and reality?
- [08:32]
A
I think so much of this is about, is about experience. Because you know the boards, the CEOs are telling their executive team, go, go, go deploy AI, but do they have the frameworks the program outlines, the project experience to actually make them successful? The answer is no, not yet. So they're being told, go make AI transformative. They're investing heavily in it. And then they wait and they wait and they wait.
- [09:05]
B
Yeah, exactly. I mean, as you say, very few enterprises have use cases in production that are creating strong, measurable roi. But the ones that are, that have done that are getting really significant operational advantages. They are they are creating competitive advantages that are durable. So, Ray, we hear a lot about AI model accuracy. We hear about the inability to do real aoi. But it seems there's another issue lurking around the corner and that's trust and governance. They're still major blockers.
- [09:44]
A
Yeah. Well, let's start with the trust so far. And by the way, it's interesting, Peter, I heard this a lot more 12 months ago than I hear it today, but it's all about can I trust the result of the AI or are they going to hallucinate? Right. And by the way, AI, large language models are still prone to hallucination and that can lead to very dangerous outcomes and even legally risky outcomes in regulated industries. So it's going to take a while for us as organizations to decide how do we build the guardrails, the AI explainability, the, you know, maybe real time insight to know what our agents are doing. So I think that is also going to delay some of the rollout and once again, the impact on white collar labor for a period of time. But I know that, you know, I'm talking about our concerns about hallucinations, but I think we also talked in a newsletter, it was about three to four weeks ago, about the reality of governance and specific regulatory requirements and governance from the governments. Can you talk a little bit about that?
- [11:03]
B
Sure. So governance is a big challenge. The frameworks for agentic AI are still taking shape because the agentic AI use cases are still taking shape. And, and so, and if you're in a regulated industry, the absence of coherent national AI regulation has created compliance uncertainty. So you don't really know how to apply. If you're in the financial services or healthcare sector, you don't really know how to apply AI to the rules and regulations that existed pre AI and are still the ones on the books.
- [11:40]
A
And not only do we have the industry regulations, we have the state specific regulations like California, New York, they sometimes conflict with some of the things coming out from the federal government. And there was a policy, I think Trump announced, what, about four or five weeks ago that federal policy, AI regulatory policy would trump all the states. But then for Fortune 1000 companies, those that have the biggest potential to benefit from implementing agentic AI, they've got to worry about sovereign multinational regulatory environments. So all those things, it's going to take a while to maneuver for organizations. So I'm going to pull this thread one more time. You keep hearing me say AI is going to be successful. I don't think it's going to be quite as Quickly, as we thought. But let's kind of. What is it? I forget what the right phrase is, but let's talk about what everyone's talking about and that is employment. What's really going on with AI and employment today.
- [12:44]
B
So this is the part we wrote about this a couple of weeks ago. We did a podcast about it. This is the part that's driving a lot of anxiety because high profile CEOs are making major AI driven decisions about their workforces. So know it's early last year Salesforce cut 4,000 people from their customer support organization globally and replaced them with their own AI agents. You know, Klarna downsized its workforce by 40% from 5,000 people to 3,000 people, deployed a whole bunch of AI and knocked the COVID off the ball from a revenue perspective. The CEO of Ford, Jim Farley, says that AI is going to replace literally half of his white collar workers. And then there's Dario Amit Day, who has more visibility into AI's near term capabilities than anyone. And he's worried that half of all entry level white collar jobs in tech, finance, law and consulting could be replaced or eliminated. And that's a little bit too doomsday. But I think it'll take some time for us to create enough new jobs because of AI to replace all the white collar jobs that are lost due to AI. So what do you think in this area?
- [14:01]
A
Well, I'll take it from a parent's perspective and that is all around some of the early indicators of how it's impacting early career jobs. My youngest son's a senior, degrees in data science and statistics. And what we're finding is that white collar job posting themselves have fallen over 16% year over year. And we're looking at some of the career office data right now and where you might have 83, 88% placement by the spring for the college graduates, it's down and I think right now the average is 23%. So real big issue. And by the way, this isn't for the people graduating with their liberal arts degrees, right? These are people graduating with software development, data analysis, graphic design. So this is some of the very what we told our kids to go study five years ago are being impacted the hardest. So my question right now isn't whether AI is going to impact jobs. It is, the question is how are we going to determine which organizations are going to thrive and which jobs will be created to replace all these lost early career positions?
- [15:26]
B
Right. Well Ray, I think we've crossed enough catharsis for our audience here now. So I think we ought to move from angst to optimism. And so we've gone through, you know, spending without returns, construction delays, slow roi, trust issues, employee impact. So why exactly are you and I still optimistic?
- [15:56]
A
Well, you know, it's funny, my metrics brother, Dave Kellogg, he said he Ray, you know why we're seeing so much noise and written about how AI is going to impact jobs? It's because it's happening to us, the white collar people on the social media, on Twitter, on LinkedIn. Because if we look at every other kind of technology driven transformation, whether that was the impact of electricity, the impact of railroads, the impact of telecommunications, the PC, the Internet, you know, shop floor automation, the, that actual disruption is actually delivered a surplus of economic output that was able to go back into the economy and actually so it didn't create a huge macroeconomic malaise. One of the differences though is AI has been and has the potential to disrupt things much faster. And this is one of those common threads I'm trying to pull through here is that's why enterprises even, even without a lot of tangible ROI, their boards and their COs are saying hey, we've got to work through this hard part of experimenting, making mistakes, maybe not driving all the roi, investing more capital than my free cash flow can support because if I can get some competitive advantage they will compound over time. And that's why they're doing. But let's start with the technology itself because it's moving pretty quick too. Peter?
- [17:35]
B
Oh, totally. The LLMs themselves are making huge leaps every quarter. So foundation model costs since the beginning of 2024 have dropped by 97%. So it was 30 bucks per $1 billion in Tokyo 2023. Sorry, $30 per million tokens at the early 23. We're sitting here now it's under $1. That's a great change in over 36 months. And it's not just cheaper, it's better. Now we have reasoning, multimodality, context, window size, all those things have improved and we have more choice in terms of generative AI models. Beginning of 2024 we had 60 of them. Most of them were open source. Now we have 650 of them at the end of last year and they're much higher performance.
- [18:34]
A
In fact, I believe it was an IBM or Deloitte report. It might have been a Deloitte report that said that the average number of models that an enterprise uses today has went up like 10 to 15% in the last year where it used to be 10, 2.6 and now it's 3.1. And I'm seeing more and more companies orchestrating their workflows to say, oh, for that particular task, use this model, but for this other task, use the much cheaper model maybe where accuracy and latency is not quite as important. So I'm seeing a lot of smarter deployments in orchestration of which models are being used. And even though we talk about, hey, only 6% of projects are driving ROI, a lot of what's being done initially has been I call it personal productivity use of AI. Hey, I'm using Claude or ChatGPT to maybe do research faster, or maybe I'm using it to do more graphics designs faster. But that end to end process workflow automation using a gentic AI, it's actually now finally starting to take off. And it's projected to grow, I think from $28 billion invested in Agentic AI in 2024 to over 127 by 2029. So that says in five years it's going to grow about 5x. And McKinsey's report says that 62% of large companies are already experimenting with AI agents.
- [20:09]
B
So the technology is getting better, faster and cheaper at the same time, which is always a good combination. And that creates opportunities for enterprises to build use cases that they couldn't have built last summer. Right, and what's producing the best results? Well, it certainly looks like it's AI native applications. They're producing great results and they share a common trait. They're not, they're not layered on old workflows, they're built fundamentally around new ones.
- [20:41]
A
Yeah, let's talk honestly. The first killer app of AI software has been with coding. The Claude code cursor Lovable ripplets. That's the first big breakout category, accounting for almost $4 billion or over 55% of departmental AI spend last year and 60. No, maybe it's 50% quickly going to 60% of developers are now using AI coding tools daily. And the teams that are using them are reporting 15% plus gains in velocity of code being released. In fact, in one of the eat your own dog food moments, anthropic now produces 100% of its code using Claude code. So one of my questions to you is beyond that killer app, are there other departments making similar type of progress using AI? Peter?
- [21:36]
B
Oh, absolutely. So go to market functions are being transformed too. So marketing platforms, especially for demand generation, for creative generation, that was a $660 million market last year and it's growing like a weed. Customer success Tools for bringing users online and supporting them. One 50 person SaaS company used AI driven lead scoring to double its win rate from 18% to 36% and add $3.2 million in ARR in six months without adding any headcount. So that goes all the way down to the bottom line. So Ray, everyone's saying vertical AI is the next big thing. I think you've got the data to support that, right?
- [22:27]
A
Well the one thing that I guess that shouldn't surprise me since my wife's a CFO in the healthcare industry, but healthcare AI is the fastest growing vertical. Now you might say, well I kind of get that, probably doing AI to read images, etc. But first of all what we need to realize is that healthcare has spent the lowest percent of revenue on it traditionally of any industry. And with AI they're spending more than any other industry, including financial services. So that's a major difference. And one of the first best use cases is using transcription. So when the doctor instead of having to take time typing in everything, they're just speaking with the patient or making notes after the patient leaves. That one company built a healthcare transcription that saved over 50,000 hours of clinician time. Now what does this mean? It means two things. Number one, the physicians can spend more time on client centric care and that's a big goal of a lot of the larger healthcare systems. And most people may not know this, but healthcare systems, hospitals especially don't make a lot of money. So if we can make the non patient time more productive for physicians, we can actually see more patients, have better outcomes and get more volume. Which is the key to trying to drive a little bit more free cash flow and margin for healthcare companies.
- [24:04]
B
Right. That's a true trifecta. Right?
- [24:06]
A
It's not just healthcare though, it's happening otherwise too. Right Peter?
- [24:10]
B
Vertical AI, legal, customer support, IT operations, cybersecurity, it's working across the board and AI models get, it's very noisy around OpenAI, anthropic, Google the hyperscalers. But these AI native and AI first companies are just grinding it out
- [24:36]
A
and
- [24:36]
B
really adding a lot of value really quickly.
- [24:39]
A
Hey Peter, this is the AI to ROI podcast. We've only got about five minutes left in today's episode. So what I'd like to do is I'd like to jump into not the fact that half of enterprise AI spend is already going towards business applications to really drive process efficacy and productivity, but there's some companies actually producing real tangible and economic return. So can you share a story about that, yeah.
- [25:09]
B
So IBM, the kings of big iron, they still support 10,000 mainframes in operation, but they were really early into tracking the performance of AI. So they saw ChatGPT come out and they said in January 2023, maybe we should figure out how AI is affecting their company. So they say that since January 2023, their internally deployed AI use cases have produced $4.5 billion in productivity savings. They've automated almost 4 million hours of work in a single year, and their average return for AI investment is $3.50 for every dollar invested. And so we're going to see more stories like that.
- [25:58]
A
MIT Nanda is not going to like this because this goes way beyond a pilot program that didn't convert into productivity. This really is tangible return on investment through enterprise scale deployment. So it shows you the potential and a great use case. And because of the competitive advantage that companies actually can gain, I don't think enterprises are going to quit investing and trying to accelerate the returns here.
- [26:28]
B
Oh no, absolutely not. The excess spending, the delays, the slow roi, enterprises are doubling down, they're not pulling back. Why do you think that is?
- [26:41]
A
Where we always used to have the fear of messing up, but the fear of messing, missing out on the potential benefit here is a real driver. Fear is a major driver. 54% of business leaders in a Mercer study said they don't think they can remain competitive in the next five years without adapting AI at scale. And McKinsey actually said that 92% of firms plan to increase their AI budgets over the next three years. So investments aren't going to slow down, they're going to continue to accelerate.
- [27:16]
B
Peter Right. If your direct competitors are deploying AI at scale and you're not, the productivity gap expands every year.
- [27:24]
A
Yeah. And one of the key things this is moving from, I did some research 12, 18 months ago and so many AI programs are being driven. Bottoms up. It was individuals going outside of company policy and using ChatGPT to help them do things on their personal work. But now AI decision making has moved from individuals or from a nice little contained IT project to C suite initiatives. In fact, I think I read like 67% of S&P 500 earnings calls or quarterly earnings calls are now mentioning something about AI. So it's really has become a C suite and board level program.
- [28:12]
B
Oh no doubt. Enterprises with digitally and AI savvy boards outperform their peers by basically 11% in return on equity, according to MIT. So they got something right there. It's not just MIT, Nanda. They did a different study.
- [28:28]
A
I was going to say it must have been another part of them.
- [28:30]
B
Yeah, it's definitely a different study.
- [28:32]
A
Hey, Peter, let's bring this home here and wrap it up. We got to wrap you up here in a minute. So you know, this is not a market that's going to be huge and produce incredible benefits because AI vendors say so. It's because we do have history and history is the best predictor of the future. Every time we've had these type of technology driven disruption, it's created amazing productivity impact, it's helped our economy. And I'm going to pull that thread one more time because of a lot of the challenges we're going to have over the next two to three years. The adoption is going to be a little bit slower than many predict. And it's going to give our organizations, our employees and our economy the chance to keep up. You want to bring this home for us?
- [29:20]
B
Sure. So let's go back to an area you and I really remember well, which is the.com era. We became senior executives during that period. But the winning enterprises in that period didn't spend the most in the construction phase. From 1995 to 2000, 2001, they built their use cases on top of what the pioneers had built before them. So Google was born pretty late into the bubble, Salesforce was born pretty late into the bubble. And they really took advantage of the knowledge that came before them and then they compounded their advantages. And so I think we're going to see some companies coming up from behind. We'll have a few model winners, but at the end of the day, I think that we're building up the infrastructure for really high speed race as we get towards the end of the decade.
- [30:18]
A
Yeah, the last two years to me with this new innovative technology has been a lot about experimentation. The next two to three years is going to be operationalization, if I can say that one big word. It's how do I operationalize agentic AI into every business process that I can say. Peter, thank you so much for being part of the conversation today. And to the audience, if you're enjoying some of these discussions, you want to dive deeper into detail. Hey, go subscribe to our newsletter and that's at AI2ROI. That's AI, the number two. Roi.substack.com Give us a review and let us know if there's any other topics you'd love to see us cover. Thanks, Peter.
- [31:00]
B
All right, see you next week.