← Everyday AI Podcast – An AI and ChatGPT Podcast
Everyday AI Podcast – An AI and ChatGPT Podcast
Ep 813: AI Cost Control 101: Why Your Chatbot Bill Is Becoming a Board-Level Problem (Start Here Series Vol 31)
00:33:57
Loading summary
Transcript5 lines
- [00:01]
A
This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.
- [00:19]
B
Most AI tools at work don't know anything about you. Every time you're stuck re explaining your projects, your team, what actually matters. Slack just fixed that. The all new Slack Bot is a personal AI agent built into Slack. And it starts with your context, your messages, files and channels. Translation. An AI that already knows how you work. Check it out@slack.com. For the past three and a half years, AI has almost felt too cheap to be real, especially for power users during early 2026. I mean, you pay a monthly fee, you open ChatGPT or Claude or Codex, Gemini, Copilot, whatever, and then you have agents run for hours completing your actual work with your contacts without paying an additional penny on top of that subscription. But that version of AI is ending, and it's going to hit even harder later today. Why? Well, that's because Anthropic's Fable 5 only has hours left until it's no longer included in payment paid subscription plans. And that's not just an Anthropic story. And it's the clearest signal yet that Frontier AI is becoming a metered utility. I mean, Microsoft GitHub moved Copilot into AI credits. Google changed Gemini limits around compute complexity, features and chat length. And Microsoft started pricing Copilot code work through credits not included subscriptions. And even neoclaud Xai's Grok has been moving users toward share weekly pools and extra usage credits. So with the explosion of agentic models that can now work for hours with expert level artifacts, the question is no longer how do we get everyone in our company using AI? The board level question is how do we use more AI without bankrupting our budget in the process? And that's exactly what we're going to be tackling today. So here is the big picture. Frontier AI has definitely now become a metered utility. And the Wild west days of using whatever model you wanted around the clock and just paying a $20 or $200 a month subscription fee and nothing more. Those days are pretty much gone. Like I said, for three years we've been doing this seemingly unlimited use of AI. And if you go back to literally back in 2023, even when the models were pre agentic, I even said eventually we're going to see $202,000 subscription plans. And I think even at that time it's still going to be a steal. And now when I throw some numbers out, you're going to see that even these 200amonth plans, or if there isn't even a plan on top of that someday, will still be a steal considering what you're going to start paying tomorrow. That's because even though the price per token has fallen by literally like 98% over the past three years, depending on what tier you're looking at, the agentic capabilities and the amount of tokens that these models are using has legit skyrocketed. And over the past two or three months, AI has quickly gone from let's use as much as we can in token max to putting strict spending limits. And sometimes, you know, developers at huge companies aren't even to use AI as much as everyday non technical people like you and me. And I think the Fable issue today anthropic fable 5 leaving subscription in a couple of hours only heightens that chatbot bill problem. So on today's show, here's what you're going to learn. You're going to learn why Fable 5 subscription exit is the cleanest and loudest warning shot for subscription AI and your team's AI strategy. You're going to know why GitHub, Copilot and Grok swapped unlimited access for pay per use meters. You're going to understand why companies are capping spend by but still trying to use more tokens and how they're doing it. And I'm going to leave you at the very end with our seven Step Playbook to control runaway AI spend. Let's get into it. Welcome to Everyday AI and our Start Here series. This is the essential podcast series to both learn the AI basics and to double down on your AI knowledge. So if this is helpful and if that's exactly what you're trying to do, make sure you go to start here series.com that is going to give you exclusive access to our inner circle community. You're going to be thrust straight into our Start Here Series space where you can go listen to every single Start Here series. The simplest way to do it on the playlist that we have there as well. You can go read about it, connect with other people who are trying to do the same thing. And if you missed our last Start Here Series episode, we talked about the desktop agent lingo, simplified going over goals, loops, plans, sub agents and how it works in Codex and Claude code. And well, if you're doing all of those things, chances are, yeah, your bills might start getting more and more expensive. Which brings us to today's episode AI Cost Control 101 and why your chatbot bill is now becoming a board level problem. So over the past couple of months we've seen this and it's not just Anthropic's Fable five. I mean, let's go back to May. Right after Google's IO conference, their big yearly conference, for the first time, Google started putting actually pretty strict restrictions on their Google Gemini. So it's actually the first time I'd ever run into Google Gemini limits on their 20 plan. Up until then it had been pretty generous and you know, after they made this shift of actually starting to count longer, you know, taking things like complexity, the model you use in chat length, all of a sudden that $20 plan that used to feel pretty great started feeling not super useful. Then in June you had Microsoft's GitHub copilot replace their premium requests with token based AI credits. So the unlimited plan was gone for Microsoft's GitHub copilot. The same thing with Copilot Cowork. Microsoft is starting to go with task level credit pricing and no longer including essentially unlimited cowork usage after they moved it to general availability. Then couple days ago we also saw Grok from XAI Space. Xai I think is technically their new name as of a couple hours ago. Right. They started rolling out the credit based system as well, which is pretty telling considering. Well, they're kind of like a NEO cloud leader now. Right. They are one of the companies that has technically extra compute and they're even charging for. And like I've already said, today is maybe the day that all of this gets thrust more squarely into focus because of anthropic fable 5. And why is that important? Well, right now at least, I mean, we'll see what happens over the next couple of hours because we've heard rumors that we might get a GPT56 from OpenAI today. Sometime between today and Thursday. But at least for the next couple of hours, Fable 5 is the most powerful AI model in the world. And at least for a couple more hours it is still available in a subscription, but after today you are going to be paying the full API price for that. So we've gone over the Fable 5 drama kind of enough, but you know, they put it out there, they had to take it away because of some problems with the U.S. government. And they brought it back a couple of days ago, but they said only included in plans through July 7th. And they said, hey, maybe one day when we have enough compute, maybe we'll bring it back, but no guarantee and why does this matter? Because like I said, for the past three and a half years, we've been able to, you know, whether you have a $20 plan, $100 Pro Max plan, $200 Pro Max. Right. But for a very low price. And I know that's all relative, Right. For many people out there, I understand a $200 a month plan, especially if you're paying it, pour it yourself, doesn't seem like the cheapest thing, but let me just show you why it actually is. So now, starting tomorrow, technically 11:59pm Pacific time, you're going to be paying $10 per million input tokens and $50 per million output tokens for Fable 5. Let me just put this into comparison now. So you probably saying, all right, what's the big deal? All right, like Jordan, if you're paying, you know, $200, you know, a month for, you know, which I am, I'm paying 200amonth for the Claude 20x Max plan. I'm paying 200amonth for THE CHAT GPT Pro plan. You know, I paid the 200amonth previously for the Google Ultra plan. So I paid for all of these. So you're saying, okay, what's the big deal, you know, if it just moves to API only, can't you just use that money for the API credits? Yeah. No. Okay, so granted, OpenAI has still ridiculously generous limits. All right, so let me put that out there. You're normally not getting that level of spend on these other plans, but I've been averaging over the past couple of weeks about 2ish billion tokens a week on the 200amonth OpenAI Codex plan. Okay. Which is included with Chad GPT. Right. So it's the same plan. So that at a, at a very generous blended rate. Right. Of a blended rate of input and output tokens. That is a $200,000 a month anthropic bill. If I were to take everything I'm doing inside of Codex and I'm only paying $200 a month, and if I were to say, in theory, move all of that into fable, that is $200,000 a month. Okay, now you see why this has been a such a big discussion in the AI world about no longer having this democratized AI access to AI. We'll see. You know what it means with OpenAI's models moving forward. They did say that they' new GPT56, kind of the different tiers, Soul Tera, Luna and you know, Soul Ultra would be included in subscription plans. So maybe OpenAI maybe one of the only few players left that has yet to, you know, kind of knock, knock out that subsidy. But that is still the reality because you have to assume that one day that's going to be the status quo and maybe OpenAI might be able to do it for another three months, three quarters, three years, we're not sure. But eventually AI is going to get more and more expensive because the models themselves in theory are going to be eating up more and more tokens. And this is exactly why we're seeing that token maxing to token efficiency shift. And we did cover that on the start here series volume 27 or episode 789. So go check that out. But we've seen it in some of the biggest companies in the world. The so as an example, we saw the very, you know, popular story. Uber reportedly burned through its 2026 AI coding budget in just four months. Tesla, we recently saw, capped employee AI tool spend to just $200 a week. And then UBS says that 60% of interviewed enterprises are throttling AI spent already. Right. So I do assume that number will probably go up if UBS does the same survey quarterly. I don't know if they do. I would assume that's go up to about 70 to 80% by the end of the third quarter because I don't think the kind of the token efficiency wall has hit the majority of enterprises just yet. So, and this is actually everywhere, right? I even had a conversation with a friend who is in software development at one of the Mag 7 companies, you know, and he said his spend is $100 a month, right. So that 200 a week at Tesla. Although when you're looking at API credits, you're like, that's like nothing. Yeah, imagine having like $100 a month. So that is the reality. So I also want to tell this to people out there who maybe don't talk to a lot of people at Fortune 100 type companies and you know, big mag 7 type companies, we are seeing actual real spend limits on the amount of AI that these employees can use. And that actually I think for SMBs and medium sized enterprises might create a strategic advantage. Right. If the biggest players are maybe, you know, having to reel in their spend and maybe part of that was their own fault for pushing these token leaderboards and all these other things. I think there is an advantage to be had for everyone else. All right, let's take a quick break for a word from our partners. Here's what most AI tools still can't do work Outside their own little box. The all new Slack bot just changed that. It's your AI teammate inside Slack and now it can read, write and act across the other apps your team already uses. No more tool switching, no more re explaining yourself every time you open a new tab. One Ops team at Engine says the summary feature alone saves them 15 to 20 minutes of use. See what Slackbot can do@slack.com okay, so why now, right? Why are we seeing this? You know your chatbot bill becoming a boardroom problem now? It sure wasn't at this time last year. It wasn't really at the end of 2025. And I think to make a very short story of the last six to eight months of AI development, if you listen to the show, it's no surprise, maybe if you're newer here, let me fill you in. Anthropic set pace at the end of 2025 with Claude Code bringing that to the desktop as well as Claude cowork in early 2026. And then I think the rest of the way has been blazed by Codex Cursor and some others. But we are now having extremely powerful AI models even for non technical work, right? When I talked about spending, you know, 2ish billion tokens a week, I'm not doing a ton of traditional coding. Yes, I'm building a bunch of programs that I use all the time. But a lot of my token spend is just doing right scheduled work that I wouldn't normally be doing right. Doing a bunch of research, creating a bunch of just documents. And the reason why we are seeing this explosion is because these capabilities are becoming more powerful and they're becoming easier for non technical people to use. So you have more and more people starting to automate their everyday work with these desktop agents that can work 24, 7 and can work for hours, right? I remember even in the earlier days in 2026 with Claude Code I would have to really push the mod to work for maybe like an hour or so. But now with the combination of sub agents and plan mode and goal and all those things that we talked about in the last Start Here series, every single day I'm usually having multiple threads running for 8 to 16 hours and creating amazing outcomes. So it's all about the amount of tokens being spent. So a Stanford Digital Economy Lab study this year showed how much more impactful and how much more token hungry today's models are than the old chatbot only non reasoning models. So essentially they found that agentic coding tasks used a thousand times more tokens than A code chat or just a normal reasoning chat. So the same task runs varied up to 30x, making forecasting brutally hard. That's the other thing. And I think that, you know, say what you want about your model of choice, but certain models aren't exactly to token efficient, right? If you look at the artifactual artificial analysis leaderboard, the anthropic models are extremely token inefficient. Yes, they are some of the best in the world. But if you are not keeping a close eye on your AI spin and you're just giving these very powerful models access to all your data, your repos, your code, you know, your code base, whatever it is, and you're giving it this big goal and then just not really watching it, right? We've seen models like fable, like opus, be 2 to 3x more token inefficient than some of their closest competitors from the other labs, right? From, you know, Google, OpenAI, you know, and then even some of the open source models as well. And then there is the rumored story, right, that a single enterprise wasn't keeping a close watch on their API bill and accidentally spent $500 million with Anthropic. I still don't know if that is true or not, but there were some reputable, you know, news organizations reporting on it. But that is the risk. And that is why this essentially AI bill has gone all the way up to the boardroom. And it used to be a couple of months ago, right? The, the message being pushed down was let's increase our seats, right? Hey, we're paying for, you know, 5,000 seats, but only 2500 are being used. We need to increase that. And now that has shifted because a lot of those seats, quote unquote, that they're paying for before now have meter billing maybe on top of a very limited, you know, monthly or weekly quota. So now, you know, there was all of this push over the end of 2025 and early, early 2026. So let's get people using this AI as much as possible, right? We all got this taste of the gateway drugs, right? The 2025 models were fantastic. The harnessing improved, the tool calling improved models reasoned and thought by default, right? So all of a sudden you had companies that were smart and doing things the right way. They showed measurable roi and they're like, let's get this thing, let's use this as much as humanly possible. And then the whiplash effect is hitting us all in the face. And that's why we have Javon's Paradox and that explains kind of this concept of cheaper, cheaper tokens, but larger bills. So if you don't know what that is, that's essentially the that as a resource becomes more efficient to use, its consumption often increases rather than decreases. So Javon's paradox means cheaper units can drive more total consumption. Simple. Right? But cheaper tokens also created those longer contexts and more agents, more retries, more automation, and sometimes more loops of tool calling in a bad way. Right. Just models calling tools when you're not looking at them and they're going over and over and over and you're maybe not monitoring that. And that's why these bills are growing, because of demand expanding faster than the unit prices fell. And also sometimes the human in the loop being a little lazy and not keeping an eye on because they're like, hey, I can give this thing a goal and give this all this data and it's going to work for 16 hours and it's going to get the job done. Maybe not looking at how much that job is actually costing or how many, you know, times a certain model is trying to go fetch a website and failing. And it's just trying over and over and over and maybe each of those tries, maybe there's hundreds, thousands of tries, and it's costing you maybe a couple of dollars every single attempt. So there's use cases already we've seen of companies maybe doing this the right way. And I think one of them is Coinbase. So what they did, and they shared about it a little bit online is they swapped their default models from essentially anthropic to GLM52 and Kimi 2.7. And they shared that, the combination of that. Right. Every single model. Right. But a lot of their, you know, orchestration, so, you know, still handing off some of the heavier lifting to maybe an anthropic model. But a lot of that, you know, middle grunt work, the orchestration, the summarization, the writing, some of those things that you maybe are fine with a, you know, one B tier model, well, they were just going open source with that and using something like Zai's GLM 5.2 or Kimi to 7. So combining that with caching, proper caching and then difficulty based routing, well, they were able to cut spend to nearly half of what it was at its peak, yet they increased their actual token usage. And in terms of output, it didn't fall off, at least according to the company. And that's also led to a surge of a new breed of model routers. Right. And what's funny Is well, I told you guys about this more than 18 months ago. That's why, especially those, you know, year end prediction series, that's why you got to listen to these and put these things into practice. Because I told you guys this was coming, right? The mixture of models and some, some other things. But we've seen a lot of, I think great innovations. Some of them like open routers, fusion, perplexity, computer computer, although I wish it was a little more afford, you know, and you've seen some other ones as well, Merge, another great example. But essentially now you have these services that are kind of plug and play and we'll do this for you. So similarly how Coinbase went in and did this all manually, right? You are having some quote unquote third party services that are going to do a lot of the heavy lifting for you, right? You enter in your API keys, you know, connect your data as you normally would via you know, easy one click, easy, you know, porting over connectors and then, well, you can just hopefully reduce your API spend. There are obviously things you have to keep in mind and do your due diligence on in terms of privacy, data sharing and all of those things. Right. I'm not going to be vouching for all of these other companies and how they do that. You have to go do your own kind of digging on those. But routing is just going to pick the model. But tuning is I think where we're going to start to see a lot of moats being built. Yes, that's right. Fine tuning models. I don't know if we'd say it's making a comeback because it didn't really go away. Right. But I think fine tuning was all the rage back in like, you know, late 2023 and 2024, before we had a handful of, you know, frontier AI labs to choose from and before, you know, bringing your data was difficult. Right. Back when, you know, we were talking about rag pipelines a lot also fine tuning models was a huge deal. And I think that again, this is something that we talked about 18 months ago, right? Several models kind of teaming up. Not just mixture of experts, right? Where you're looking at sparse versus dense models, in giving a query to a big model in a big model, just activating the, you know, the experts that it needs within those parameters that it carries. No, I think we're looking at the mixture of models as being something that is going to become increasingly popular in the near future and I actually hope that we see that from the actual Frontier Labs themselves. I Would love to see that from you know, anthropic, OpenAI, Microsoft, Google, etc, right? Where even within something like Claude Desktop or you know, Codex where you can go into a certain mode and it will do it for you. We've kind of seen previews of this that I don't think were super well done where you had these auto modes and it decided how much reasoning it would want to use. But that was really a little bit more at least for our non developers, non technical people. You know it was more of I think just an experiment to maybe you know keep compute in check around new model launches. But I do think that the combination of mixture of models and fine tuning models is going to help companies not just keep spend in check but actually build a moat. So this kind of flew under the radar. You know I saw this when I was out at Microsoft Build conference. But now Microsoft Foundry offers fine tuning as a service, right? So as an example, you know Foundry has a ton of models from all of these different providers, right. If you just need one model and if it's 80% of your usage, let's just say as an example is doing some financial modeling around a certain sector instead of taking one of the big state of the art trillion parameter models, right. You can work with Now Microsoft in Foundry they do this as a service. They can find the right maybe a smaller or medium sized model and then just fine tune that or essentially create a version of the of a medium sized model that's more cost efficient, faster and well just better because they're essentially going to train a version of the model just around your specific tasks. Speaking of that, that seems to be a key area where Thinking Machines Labs is starting to play, right? Former you know OpenAI CTO&CO founder Mira Muradi with Thinking Machines Lab. They just came out with a study that showed Bridgewater and Thinking Machines Lab reported a tuned specialist beat a frontier model at about 14x lower cost. So that does seem something that Thinking Machines is doing as well. Essentially offering fine tuning API as a service. But when you have the big labs, right and sure we'll throw Thinking Machines in as a big lab in terms of market cap valuation, money raised, right? You they're at least in that consideration Microsoft obviously one of the biggest and best in the country. Sorry in the world, right. When you are starting to see fine tuning as a service that is not about model specialization, that is about the the in pending reality of subscription models going away in enterprise companies having to look at that, right? If you were paying for as an example, if you were paying for 10,000 licenses of Microsoft, GitHub, Copilot, and then all of a sudden they switched over to usage billing, right? Are you going to continue to do that? I don't know. Right. This is where you're going to have to start looking at these alternatives. So here is your Monday move. You have to build the spend router, right? Whether you look at some of those other services, whether you start doing this and duct taping it yourself internally, or whether you are just preparing for when the big labs offer this, you have to start putting this in place. You don't just cut AI blindly. That is the absolute worst thing you can do. And unfortunately a lot of companies are doing this. So don't do that. Right. You have to start classifying work by value, risk and complexity and start using some of these cheaper open source models again, go through all your due diligence on the privacy data, you know, IP side, all that, right? But you should be looking at these tuned specialists, routers, CAPS and frontier escalation, because the winners ultimately are going to be the ones I think like Coinbase, that are finding ways to actually spend more tokens and to use more tokens. I don't think there was anything wrong per se with the token maxing era, but it was more of doing it irresponsibly on the API side and using the most expensive models. Right. I do think that you have to have some experimentation on these models that are a fraction of the cost and you still get maybe, you know, 85 to 95% of the performance of the state of the art models. So here is your seven step playbook to control runaway AI spend as we wrap today's show. Number one, you have to meter first. So you have to get token spend visibility by team, task and model. Right? So similarly how I said, hey, right now I'm going through personally 2.5 billion tokens a week, I can break that down by project. I actually build a tool inside of Codex that helps me do that. Right. You can be doing these things as well. You have to start seeing where your tokens are going to, who is spending them, what task, what model and what are the outcomes as well. Number two, set budgets. After you do that, you have to be able to cap spend per team user and workflow with overage alerts. That doesn't mean you cut use, right? So let's say you first meter and you say, here's what we're doing on our subscriptions. What is that? What is that spend that you're going to have to do to get the same outcomes. Right. So don't cut budget without protecting your outcomes. All right. Number three, you have to start learning to default to cheap. All right. I'm not saying the cheapest models but now we do have models that are extremely capable open weight models like GLM 5.2, you know, like Kimmy 27. Right. We're gonna I'm sure have some other things from Quinn. They have some great models Deepseek. I'm sure we'll come up with another good one. Again, go through all your proper IP privacy security but you have to start looking at these maybe open source open weight models whether you're running them locally or paying for them on the API side, they're normally a fraction of the price as the closed frontier models. Number four, you have to route by difficulty. You shouldn't be as an example having Fable 5 on Extra Ultra Reasoning, whatever it's called inside cloud code. You shouldn't be doing that to help you write better emails. Right. You have to send the right type of tasks to the right models. Then you need to understand your get proper caching. You need to trim and reuse repeated prompts and start lean sessions per task. Right. Even simple things like if you're working on the front end using things like projects. Right. Certain, you know, certain providers do things better than others. Right. As an example, Codex is great at compaction Auto compaction. It's really good. You know Quad recommends as chats get longer to start new chats to save on cost but you have to start doing those kind of non technical best practices as well as caching and trimming. Number six, fine tune the repeats. Yes. If especially I think if you are a Microsoft organization, I do assume that we're going to start seeing fine tuning as a service. Right. I think thinking Machine Labs and Microsoft offering this leads me to believe we're going to see this probably within nine months from all of the big players. You have to start looking at and understanding what it's what are we going to fine tune, right. What type of model is going to do 80% of our AI use for certain teams? And well if you fine tune a model it might be a little bit of work and a little bit of cost up front but you have to be able to math the math because chances are it's going to save you a lot in the long run and then last but not least, escalate on purpose. Call the frontier model only to close hard problems, especially if that frontier model is not included in a subscription. And I say that as a very timely reminder. Yes, Fables Fable 5. Couple more hours and it's gone. We'll see what happens with OpenAI GPT 5, 6, Seoul and Soul Ultra. They did say it's going to be included inside Codex, so so who knows what's happening. But all I know right now as we wrap up today's show, the future of AI is token efficiency. And it is no longer if you are the decision maker. Six months ago, regardless of what tool, what system, the right move was use as much AI as possible. And that still may be the case, especially if you're an OpenAI organization, because they are pretty much the only player right now that hasn't drastically either cut the subscription limits or that hasn't gone to true metered access. But chances are some of your team, some of your organization's most needed, most used AI tools have converted to metered pricing. So that doesn't mean stop. That just means be smarter, don't stop experimenting and well follow us daily because we're going to be helping you every step along the way. I hope this one was helpful. If so, please make sure to go to starthereseries.com go sign up for our community, Go network with other people who are leaders in the AI industry, figuring it all out there. Also, if this was helpful, please subscribe to the show on Spotify and on Apple Podcasts. Thank you for tuning in. Hope to see you back tomorrow and every day for more everyday AI. Thanks y'.
- [34:07]
A
All.
- [34:12]
B
Here's the thing about AI at work. You can't use what you don't trust. The all new Slack bot runs inside Slack security boundary, only sees what you've already allowed it to see and never trains on your data. Personal AI agent with the trust who actually put it to work. Slackbot from Slack. Learn more@slack.com up.
- [34:38]
A
And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everyday AI.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.