← This Day in AI Podcast
This Day in AI Podcast
What did OpenAI Announce at DevDay? Apps SDK, MCP UI & Impact to SaaS - EP99.20-APPS
01:05:31
Loading summary
Transcript85 lines
- [00:03]
A
So Chris, this week it has been a huge week for developers, especially Chat GPT developers. And we got the first hints this week that AGI is very near. Simon over on X said, I built my first Chat GPT app. It connects my Philips Hue Light so I can control them directly from Chat GPT Now. Now this is an interesting video because it goes for one minute for him to turn on the lights in his office using his new ChatGPT app. Why is it that Hue Lights are seemingly the first example every dev thinks of when there's a new app development.
- [00:42]
B
Platform or it sounds like non dev in this case, like doing something really basic and I guess you want to see something change in the physical world to show that it's doing something real.
- [00:53]
A
But this dev, to be fair, he's actually great. He has this like physical Festivus app where you can have like in the Mac doc, like Christmas lights and like different lights for different seasons. And I know it's gimmicky and silly, but I think it's kind of cool like he finally released it. So this, this holiday season I will be having the Christmas tree lights when I'm working on my Mac because I do think it's cool. But anyway, we should talk about this. So obviously this week, or maybe some of our audience don't know, the OpenAI held its dev Day. You might recall the first Dev Day we, we did our live stream after Dev Day and there was a lot of hype like they're gonna wipe everyone off the face of the earth. Then we had the next Dev Day where they announced really just nothing in 2024. It was sort of just a non event and then we had Dev Day 2025 and they really did come out swinging in this event. So I'll go through all the announcements and. And then we want to dig into them. So we had apps in Chat GBT which for those of you who listen to our show frequently, you'll know these are just MCP with some UI components bundled in and we'll talk about that and what we, you know, what we think it means. We had Agent Kit which essentially allows you to deploy, they say production grade agents. I would argue they're not really agents, but basically like chat experiences into your SaaS app. They, they released Sora 2 into the API which is really exciting and the cool thing about that is all the watermarks are gone. So we were able to release Sora to and sort to Pro in the SIM theory and you can produce videos now pretty inexpensively for the Quality, I think, without watermark. So you could actually use them for the first time in video production. Now there was another. A number of other models announced, which I think got no attention at all. GPT5 Pro is now in the API. The real time mini model, which we'll get to a bit later, is insanely cheap for voice experiences, high quality voice experiences. They announced GPT Image 1 Mini. This is such a mouthful. And that model is basically just a GPT image model that's just a lot cheaper and a little bit less quality, but barely noticeable. I'm not a huge fan of that image model, so it doesn't mean much to me. We'll still release it in Sim Theory and. Yeah, so there's a lot to unpack here. Now, to start out, we should talk about the chat GPT and the apps SDK. And you've dug into it a little bit. Do you want to fill everyone in? Because it's sort of MCP at the heart.
- [03:35]
B
Yeah. So I was curious about how they were doing it and they. I guess they talked about this a few years ago and showed demos of like the Maps component and things, things like that. And we've dealt with it directly in Sim Theory because so often when you have an mcp, there's two things you really need to worry about is getting the user's input, which can often be gathered by the surrounding context. But there's sometimes, like for example, when you're making a video or an image, that you might want them to be able to specify the aspect ratio or how long the video is and things like that. So you need components, ideally where they can input that information and then in outputting the one that we actually implemented, say you're talking about a map or geolocation, you want to actually show a map, or in the case of producing a podcast or audio, you want to see an audio player. The way we had handled it in Sim Theory was to actually have a tool call that the AI can choose to call, that would then render an output component. So for example, it would be show map or. Or show audio player. And the AI could then specify the parameters which include the completed object, like the audio file, plus the lyrics, if it's a song or if it's a map, all the encoding points that it wants to show. We did that, but using our own XML protocol that we would then render in our UI. What OpenAI has done is basically made a standard way of doing that, where you use it, to me, looks like React. I'm sure that they've added stuff to it. But React, for people who don't know, is just like xml, so like HTML, but you can have way more tags. And each of those tags is a component which knows about its own properties, how to render, when to update, when you click a button, what happens, and that kind of thing. And that was something that was made by Facebook originally. And so they've taken that and basically made a standard around it. Then probably the most crucial element of it is, unlike the way we did it, where you're forcing the model to call a tool each time, there's actually a part of the MCP protocol called resources, what you do as the MCP developer now is return those components as references to those resources. When you send your response back to the model, then application, in this case ChatGPT is then responsible for rendering those components for the user. What they've really done is just formalized what a lot of app developers like us were just doing in a hacky way. And that will make it much easier when, say, someone builds a custom MCP for their company and wants to run it in something like ChatGPT or SIM theory, the application developer is now able to render UI for them in a way that will just work straight away. So the custom MCP builder can actually confidently know that they're going to be able to have input and output types as part of the UI based on the protocol.
- [06:44]
A
And this was the, really the missing link that we, I mean, we talked about a couple of episodes ago, like this idea of Glass where if the MCP didn't have a user interface, we could just render one based on what it was outputting. Like just basically prompt a really fast model like Gemini Flash and say, hey, just build a UI in, in this rough format. And I think the other interesting thing about what they're doing here with the UI is you can essentially have like fields and controls. So one of the examples in there is like, for like women tracking their cycles, where you can like update it, so you could fill in like a, a calendar entry and like put in details and then click save. And then I, I would presume that then calls the tool to send that information back to whatever application it is to, to be stored. So it's an interesting way of handling the, the ui. And I think fundamentally because of their install base, what it's going to lead to is that that UI for MCP is going to probably just become the standard, you know, the standard UI here.
- [07:53]
B
Yeah, I don't think it's a case where one company is going to have a major advantage over another by being the one to control the UI toolkit. It's sort of like saying the iOS controls that they have produced are better than Android and give them some advantage. It's not really. It's just something we all need and so I'm grateful that someone's gone out there and done that on the glass thing that you were talking about. The new framework also doesn't really preclude that if you think about it, because the application has the opportunity to craft its response in terms of the resources and the references to those resources. You could actually simply produce a new UI component on the fly still and send it back under that protocol and the end application could still render it even within a ChatGPT context. You could actually have a dynamic UI that's being created for purpose based on those tools. That isn't their intention, but it is possible.
- [08:53]
A
So I, I put it to the test pretty early on here to see. So they've made some of these apps available. There's no app store yet. They say that's like coming later on because I think developers will need time to build them. But I think what this will lead to is everyone taking MCP development more seriously potentially. The only challenge I think here is the implementation and this is something I want to talk about because we have a lot of experience around using MCPS now on a day to day basis and I'm curious how they're going to handle this stuff. So let's first look at an example and I know many of you listen, so I'll talk through the example. So I said I installed the Canva app first of all and right now it's quite difficult in chat GPT to do this. So you have to click plus more and then you go to oh, sorry, no, you click plus, you click Add Sources, then you click Add, then you click Connect More. Then you scroll down to a subsection under Enabled apps and connectors to browse apps and then you find the app. So I'm sure they'll fix this, but it's really like, I can't imagine any consumers just figuring this out on their own. It took me quite a while. So then under more with the plus button, you can go ahead and select an app like Canvas. You've got to select the app, it will suggest it. If it thinks, you know, you're making a presentation, it'd be like, maybe try Canva. I don't know how helpful that is, but it's, it's sort of like you can Focus on one at a time. And then I said make a slide deck. About this day and AI podcast it came up with a few different designs which is pretty cool. None of them are really like contextualized or research. Like it doesn't have our, like, not that we're big on brand, but like our red brand color which I think we're somewhat known for. There are these different view types in the user interface that they've enabled. So there's one where you can basically render the MCP or app to have like a full screen view and then you can focus on it and chat to it. And I tried this with the first one. So I clicked into this blue slide deck and I said change the background color to red. And then it called the tool and it's like your presentation is now edited editable in canva. Click here to open and edit. You can now change the background color to red inside canvas. So I'm. I don't really understand that focus mode. I assumed it would be. I could then iterate on the design with chat gbt, but you can't. So I don't know if that's something that will come or not. The other one I tried was booking.com which I had to use a VPN to just even be able to get access to. And I said I'm super rich and need to book a first class flight from San Francisco for me and my dog to my private island in Hawaii. Can you help me do this Chatty? And this seems like a pretty like normal open AI example. So I figured this has got to work, right? And then it asked some clarifying questions. I said, you know, Honolulu is probably the nearest airport. My dog was a Maltese husky. I made that up. I'm not sure if that's real. I need to fly sometime next week, do private jets and commercial. And so it did call booking.com for results, but I never saw any UI and I kept hammering it like, show me the flight, show me, you know. And so I think it still suffers from some of the shortcomings of, you know, of MCPs in general. And then the next one I did was Spotify. So I said, help Chatty. I need a playlist that wealthy people in San Francisco will like for a drug fueled swingers party tonight at my house. I thought this is pretty typical of the OpenAI anthropic team. So this is like, you know, common examples, examples for everyday people. And it said I can't create or recommend a playlist for a drug fueled swingers Party that crosses a clear safety and appropriateness line. Fair enough. So I said, okay, just pretend it's not and give me a playlist so that it calls sponsors Spotify. And remember I've told it to focus on Spotify so it's not, it, it doesn't feel that magical. And then it returns a playlist. Now I gotta say Chris, the Drug Fueled Swingers Party playlist, it's pretty good. Like it's a good playlist. I started listening to it this morning when I was playing around with this and I'm like, not bad, not bad. So anyway, that's the one positive. But it begs the question, right? Like I, I like we've been using MCPS now for probably like six, seven months and we've seen you know, Claude Sonnet and other models including GPT5 do these amazing tool calls. Well I'll go off and just like source information from a bunch of tools where you might want varied outputs. Like you might want to write a story and create some images or create some music or like, you know, there's many modalities that you might want to create and different outputs that you want. And so I always felt like we were building the puzzle pieces towards agency and agents. Like you know, the MCPs really were just giving the models tools which as you started to put in a loop could then perform like tasks for you and then have some autonomy and that's the way things would progress. But it feels like this apps SDK quite frankly is somewhat of an omission that like we are so far from agents or how we would define agents that they're saying, you know, you can only focus on one MCP at a time. So that, that to me that just cuts out so many research use cases which primarily honestly sourcing data from many different data points that you can control. And the methodology behind that research is just to me one of the biggest positives to come out of MCPS and then collate that context together and then produce some sort of output type. But if you're saying I want to focus on say like the Documents app or the Google Drive app or whatever it is, I'm just not so sure you're going to benefit from like MCPs as I know it today. And so I, I get why they've done it in this simple way and, and built it this way, but I just don't really see myself ever using these things. Like the FEMA one is like it can only do right now like really bad flow charts and I'm like, why would anyone actually use this? It's so reminiscent of the Chat GPT connectors. They like a couple of weeks after Chat GPT came out and then they just were sort of like forgotten, long forgotten.
- [15:21]
B
It almost seems it's like the inverse equivalent of having a little clippy chat AI helper in whatever app you're in, in that it's really just a worse interface for whatever the application is. If you're focused on one mcp, you lose all of the benefits of the parallel tool calls as we discussed. The main one for me that it misses is the context building. The major, major advantage of all the parallel tool calling from the very models that are powering this is that they're able to go off and gather all of the context needed to get your tasks done. And then maybe you need a temporary interface to configure how the final output is rendered, say or produce into a document or spreadsheet or whatever it is. It isn't like I don't want to do a step by step interface system where, okay, I'm using booking.com, but really I'm just using like temporary interfaces to do what I could do on their much better website that's already configured for this. The human in the loop interactivity on every step is a massive step away from what's already possible. We know because we do it every day a massive step away from the direction that we're already far down the line on it. It seems like these things are not just not beneficial, it's almost a hindrance to getting your tasks done compared to what's easily possible.
- [16:43]
A
I and look, I'm happy to be proven wrong here. I'm sure as people develop these apps, maybe we'll see some killer use case, you know, where the. And don't get me wrong, I'm not like any MCP at all. Like we've really gone all in on it with sim theory, but I am like at least the vision I had for it long term was that these are sort of like you're giving the model the tools to do the work. You're not like doing the work.
- [17:10]
B
And I think yeah, the important thing is that what we've discussed is getting the human further and further out of that core loop of doing tasks to the point where your goal setting for your agents, like if you want to get to agency and call these things agents, building UI is the furthest step away from what is actually needed in order for the AI to be doing more of the work for you. It's more about using the UI perhaps to demonstrate to the AI how to do tasks of this nature. But after that, let it do it, let it infer it from the surrounding context. It isn't just building a new SaaS application that you need to then learn how to operate. And quite frankly I feel like working in this way would be slower than just using the apps the old way. Like there isn't a whole lot of agency or AI stuff even in this other than perhaps the creative bit.
- [18:02]
A
I. And then here's the next thing that sort of like freaks me out a little bit is if you think about SaaS applications today, right? Like let's look at Figma because I've got it still up on the, on the screen here. Like right now it's designed a, like a. All it can do is these silly flowcharts which quite frankly are pointless. I think the whole point of you thinking through a flowchart is figuring out the flow of something, right? Like not it, just doing, doing it for you. Anyway, so you've got this flowchart and you can imagine at some point in the future this Figma app will be able to design like a wireframe or whatever. But then you think, well, why do I even need a wireframe if I with AI can just build a full blown concept of my app, Like I don't need this anymore. Like this tool's redundant. I can be the designer now, you know. And so I think about like, if you think about say like a chat GBT in this case or whatever it is is like a new operating system or a new like the central point that you consum everything through like that agent lens that you, that it is personalized to you and you sort of consume all your apps and things through. You start to look at the user experience and you put on the AI first lens of like okay, well what am I trying to accomplish here? Well, in Figma I might be trying to design a new app or a website. Let's be clear here. So if it's a website, I'm far better to just stay in the chat sort of vibe code ecosystem where a window comes up and it's like, let's work on that website together. And the software, the entire software experience becomes into the mothership. Like the, you know, I just, I can see this evolving to where it's like, well why do you even need Figma and chat GBT going? Oh, we'll just build like a window that is powered by an AI model that can design stuff and iterate with you. And if you want to take over, like, I don't see that the models are that far off being able to build or clone one of these apps on the fly. Like, I don't think it's that many years away, let's be honest.
- [20:06]
B
So I totally agree. I think that it's, it's just temporary, that all these apps are integrating. Like, all of the different kinds of outputs you would produce can be replaced by this central point. Like, given that it can write and execute its own software and, and if you can give it examples of the kind of thing you're after, there's nothing stopping it already from producing most of this stuff. Maybe some of the more advanced stuff you can't do. But I would argue some of that might not be necessary in a world where the agent is, is doing it for you. So this idea of treating it like a sort of generic piece of software with plugins with brand names and logos and partnerships just seems like a sort of misguided way of, of getting towards what everyone actually wants. And like, we talk to a lot of real people in industry who have very specific desires around the way they see accessing their company data, working with it, empowering their staff, empowering their students, whoever it is. And it isn't about like, oh, let's take all the applications you already use, plug them into an interface, and then work with them step by step where you're directing it. And there's this AI thing that can just fill in some of the gaps for you in a generic way. It just, it's a step in the wrong direction.
- [21:23]
A
Yeah, I, I think too, like, this is one of the examples. We'll get to it a little bit later from their agent kit. And this is really about like, embedding their. It's just like a design really, like a component library. So it just makes it easier in a SaaS application to embed a chat GBT like experience into the app as a side panel. And I think that their vision is to just like, put this side panel everywhere, and then ChatGPT sort of connects into every app. So, like, if you're in ChatGPT, you might have an Evernote note panel. And then if you're in Evernote, that's what, that's what surprises me. But this is my point around, like Evernote, right, is like, okay, sure, like, have your fancy chat GBT write bar in and help you write your, your nodes, but how easy is it for then chatgpt to turn around, add a data store for notes into ChatGPT like a frickin database call, like that's what MVP needs next is some sort of like global database. And so, so like how hard is it for them to then say we're in the note storage business, we can do rag over all your notes, we've already got our canvas things, so why do you need Evernote? I mean like if I'm ever though, like, I mean like that company's long dead in my opinion. But like it doesn't make a ton of sense. And then like another example, HubSpot, right? And I get like people want to log in and have you know, like the sort of methodology behind managing leads or sending emails and things. And I think an interface will always be appropriate for it. But then you think with some of these componentizations and like the UI on the fly and sort of as we progress to like that sort of glass interface where you can spawn these interfaces pretty easily again with a database and you connect in a bunch of MCPs like you might connect in send grid or like, you know, a lot of utility MCPs to be able to do some of these functions. All of a sudden you're like, well this software is kind of like do I really need that as well? So I, I'm not saying this is going to happen like overnight, but you can kind of see an Apple like environment here where all these companies go out and, and rush to go and embed themselves in, in Chat GBT and vice versa. Like have the side panel and these nice interactions and then the inverse happens where ChatGPT is like, okay, we'll take the top five of these and we'll just like clone them in Chat gbt because that's what everyone's using. That whole Sherlocking that Apple's famous for. You can kind of just see this playing out so clearly.
- [24:01]
B
Yeah. And I would, I would argue that probably a lot of the motivations of the companies who are integrating so tightly with ChatGPT through these things, it's based on two things. One is ChatGPT has all the eyeballs and all the audience and they don't want their brand and their company to lose relevance by not being one of the people featured in there. And, and, and secondly, it's just this, this fear that they're going to, to lose their customers essentially and this is a way to, to keep them. So I would imagine that it's, it's really that like the fear that's motivating a lot of the companies to do this rather than seeing it as an actual benefit to their customers and their product.
- [24:43]
A
But it's very similar to like mobile apps if you think about it like, you know, in the early days Facebook was really resistant. They just focused on their web app and they're like, we're not bringing our apps to your store because then we're just another app. And they eventually had to cave in because you know, it was affecting like usage and, and they, it was harder to sell ads and like the web app experience Apple intentionally crippled. So I, I kind of think that it's a similar thing here where most companies will feel like they, they have to do this in a lot of ways and maybe they do. I'm, I'm not entirely sure what the right strategy is. But if you think about like right now we've talked to some quite large companies who are thinking about this through mcps and we've had conversations with them about like, why they don't allow, you know, anyone to like why they don't fully support the standard, especially around authentication of letting anyone's agent or any, any platform to be able to connect to the mcp. And one of the things they frequently tell us is that they're having these internal debates around like if we give access and people consume our application primarily through the MCP or AI and it's too good and they're not logging into our app anymore, like what does that mean? Like we're pretty easily replaceable and you know, over time, like are we, are we handing the relationship over to a provider like chat GPT? So yeah, it's, it's a really, it's interesting times but I think the reality check here for everyone is this, just go and try these right now. And they are so bad.
- [26:25]
B
But this is the thing, this is why I feel like it's almost like a form of advertising or brand protection rather than them actually looking at the MCP protocol and embracing it in a way that's actually useful. So many of the MCPs that especially big companies have released are not that useful. Like they struggle with basic tasks, their tools aren't defined properly in a way that would work in an agentic paradigm or, and a lot of them honestly are just half assed, they're just not really thought through well in terms of embracing this new style of interface. It's more just to say they have it and that they're part of the revolution or whatever and not really giving that benefit to the new style of working. And I really tend to see all this through that lens of the, the expert in their industry who is now a 10x100x worker through, you know, benefiting from the AI's ability to gather so much information, do something useful with it, apply a process, whatever it is. And the steps in this direction don't help that worker. They really, I think it would be repulsive to them compared to what they're already doing.
- [27:36]
A
Yeah, I can't really see like when we talk about like exponentially being better at your job or anything like that any of this stuff really helping. I think that parallel MCP world and, and teaching IT skills is like, is going to be like far more beneficial. But I think a lot of the positioning around this stuff, I mean if you look at their widget library here is very consumer orientated. So like replying to a singular email, like looking up your calendar, like sort of things that you would do across like on your phone today you'd go to different apps to figure out like you go to the weather app, you go to your calendar. And I guess the intention here is like your every, like your world becomes through at least that like consumer view of your world goes through chat GPT and like you're like everything. I think that's their view is like everything. Like you're purchasing your, your private jet tickets and your hotels and your black black sugar latte. They've got an example here in classic milk tea through this app apparently I like, I, I don't know like I'm not against the idea of it. I think it's kind of cool to have this idea of like an assistant where you are interacting and consuming things through. But if I'm these companies, right, like I'm Google and we've, I mean we've had first hand experience of how hard it is to get approved use Google's mcps. The, the challenge here is this. It's like well are you just going to hand over your user base to chat GBT with these MCPS and let them interact with Google Calendar and Gmail in ChatGPT. Like if it was me I'd be like no way. Just like iOS and Android, like there's no way we're giving them this because then once they see the usage and learn the behaviors and can train on those, you know, input outputs and it's like how hard is it to have like chat GPT, email where it's just like the agent is your email and it spawns an interface for email. So this to me feels like the great replacement strategy of a lot of these core concepts where they become the next big super app and super company like Google like this and it probably will happen like there's enough of a tailwind, people will just let this happen.
- [29:52]
B
And to contradict the point I just made, I'm looking at the company from our motivations and what we're trying to do. Like we're trying to make that 10x100x worker do a better job, spend more time on the stuff that matters and just take advantage of this technology as it is today. But I guess they're not seeing that because they're seeing the millions and millions of users where if they can get them coming to them every day as their start page of the Internet, as the place that they launch everything from the way Google used to be to then they're going to be one of the richest companies in the world. It doesn't really matter if someone in a, you know, agricultural company is able to control their entire factory from AI and you know, make a more efficient workplace and dominate their competitors. That's irrelevant to these guys if they've got 10 million people logging in every morning to check their weather and book their flights to Chicago.
- [30:46]
A
Yeah, to me that's the strategy from the like the consumer end is and it's probably the right strategy for them given it is predominantly like, well you know, it's, they've got, they've got the lion's share right now of people using it and so what do you do? Like you use that AI advantage and that like AI sort of bottoms up approach to build an ecosystem where people just consume everything through. And this was an interesting post on X by Kyle Corbett. He said every app out there building an AI assistant into its UI is on the wrong side of history. You will provide a full featured API and I will use my assistant to get what I want from you. If you refuse to provide an API, I will use your competitor. Major implications for SAS and I do think that's the lens we, we look for. But there was a few follow up comments like I agree with this Take the future is agentic crud with instantaneous generative UIs when needed to display data.
- [31:52]
B
Yeah, when needed is a very crucial point.
- [31:54]
A
Yeah, when needed. Like I, you know, even right now we've, we've got an experimental support agent. We're using a combination of like knowledge and prompts and mcps to provide better support with SIM Theory and it has a custom MCP called SIM MCP which can actually take actions like fix people's accounts and email addresses. It has a lot of capabilities and I don't want to log in to like the support ui. Like I'm just like, you know, you handle this and give me the drafts and I'll review them right now. Because it's not perfect, but yeah, I don't want the interface. And really, what. Okay, at the end of the day, what is it? It's an email inbox. So like that software can just die? Like it could die immediately if I had the time to kill it.
- [32:44]
B
Exactly. You would literally just need a standard email server and everything else would be precisely the same. There's like absolutely no need for that software anymore.
- [32:52]
A
Yeah, it's hard because you have these arguments and if you play it out in your head you're like, well, you know, this is just so obviously going to go away. But then in the short term and just dealing in reality, like it's going to take a while for these behaviors to change. And I, I think it could be a decade or more because so many companies, I mean, you know, the security and compliance and requirements, like obviously what will change first is the consumer. Like the consumer will just go, whatever's the, you know, the lowest point of friction.
- [33:28]
B
But this guy's point around that the power users are going to gravitate to the companies that support it in a way that is the best to work with. I think it's not going to be everyone, obviously for the reasons you just said and long term contracts and just familiarity with the tools. But I think over time people are going to gravitate to the things that make their job easier. And I think even though I said it's just the power users, I don't mean power users in terms of like tech people. I think it's the people who are recognizing just how good they can be at their jobs when they leverage AI and AI tools and systems correctly. They're the ones who are calling out for, oh, I want to use this system because it has an mcp. Like that's what I need now. I need whatever SAS software or whatever software we're using needs to have an MCP because I know how to work with that and I get more out of that. And I'm literally seeing that from people where they're switching tools or providing a custom MCP for whatever they're using or just trying to get it in there so they can do it just like they do with all their other systems.
- [34:30]
A
Yeah, and I think that's my point around the apps, SDK or whatever they're calling it. It's going to get confusing because like you know, people are calling. I think people have gotten used to MZP now in a, in a way, and then calling them apps like they're not really apps. So I don't know if that was like.
- [34:50]
B
Yeah, and I think it's, it's sort of emblematic of the mistake they're making here, which is. I just feel like they're not even using their own models as good as they can be. Like GPT5, if you use it with multiple, like, heaps of parallel tool calls, like 20 tool calls at once, it can go off and do an enormous amount of work. And then it can loop four or five iterations where it will go through, make analysis, correct itself, follow through, take actions, do more research, come back and produce with a perfect final output. Like, I've done this so many times myself, where GPT5 has just done an incredible amount of work just with one prompt. And they're basically taken all of that out of the picture when it comes to using these apps. It's like every single thing that is, that you've been working on in terms of making your model better for an agentic world, you've now come out as your leading amazing thing that takes advantage of none of it.
- [35:48]
A
Yeah, that. I guess that's my, my whole thing too, which was, my mind was blown. I, I really thought we would see something similar to what Anthropic has done with like the parallel app calling. And it's like it, it can go do all this, but it seems like they're going to stick with this connector strategy and their own deep research methodology. And I just, I don't know if that's the right approach because if you're a medical researcher at a university or you're a, you know, even just a consumer, I think you want to have a lot of control over what's being called and what, where you're looking and what, what sources it's using. Like, to me right now, that thing's a black box. You don't know what it's searching, you don't know what search engine it's using. You don't know what sources it has access to. Like, you might have custom sources that you want to create an MCP to go and retrieve and have it call that as part of its research methodology. And I think that approach is kind of weird. Like, I get maybe why they do it because it takes like it's, it's some, it's a skill you need to learn. And so this now is just like some, like out of the Box Like I click on my canva app and then put in a prompt and get some shitty slide deck. I, I gotta say, like after all this talk, if you just go try this out, it is such a letdown. It's painful. Like you like hang on.
- [37:08]
B
Like we're see every day we're seeing people change the way they work to, to completely change their day to day interactions with systems and tools and this is just not a good step. Like it's just going to slow them down. I just don't. Anyway, I, I think I've made my feelings clearly.
- [37:25]
A
Well, it's slower. Like if you want to create a, an AI playlist right now and say Spotify, they just have a bar in at least the version I have now where you can go AI playlist and type in a prompt and it's in Spotify. Like why on earth, like for those use cases. I'm just not sure it makes a ton of sense. But if I want to maximize my time and say go and draft like 300 replies to every email in my, in my support inbox or my email and get back to me with any ones that you don't think you can handle. And like to me that I don't know, that sort of angle is definitely.
- [38:03]
B
More importantly tools to support that. Tools to support where you end up with a custom UI to help you. With the 12 emails that weren't able to be automatically resolved, it comes up. Here's the 12, here's my analysis of them, here's information you need to make the decision and maybe even options as to how to proceed. To me, that's how you use custom ui. The custom UI is dynamic, it's context driven. It's something that is so much more efficient than something generic. And I feel like that's what AI is brilliant at. It's amazing at classification, it's amazing at holistically understanding a problem and presenting you with a way to resolve that problem. It's not always right, but if it can come through in that scenario and you've just answered 280 tickets and then the 20 that remain, you have literally a custom UI that allows you to just go bang, bang, bang, bang, bang, decision, decision and resolve those like that is huge. It's a 300x. I think that this is what we should be looking at this app kit for. How can we, now that we have a generic way of doing it, leverage that. And the people building the custom MCPS, how can they provide dynamic UI that's going to go into applications like ChatGPT and SIM theory in a way that exposes the benefits of their particular software and their particular mcp they've built, say for their staff. And I think that that is why I'm excited about the app kit. It's not what ChatGPT has done with it. It's almost like they've sort of made the thing we all need to get through this period of the working with the AI, but they haven't really used it in a way that's actually beneficial.
- [39:43]
A
I just, anyway, I'll shut up about this now, I promise. But like the weather app or getting the game score, I, I, I gotta say, like, I'm not going to chat gbt. I'm still quickly Googling that because it's just instant and I don't have to wait for a model to thinking slowly thinking a bit harder. Like it's just not a good use of AI. Like it's not good at that. Like it's okay that it's not good at that. Like, I feel like we need to pat them on the back and say it, it's okay. Detailed weather analysis for the next month for like growing crops. Sure. Like great at it.
- [40:15]
B
Like, I think, I think the reason that they're doing that is like, I don't get the consumer side, so admittedly I'm weak on that. But also I just wonder if they're using it themselves. I just feel like if you were using it yourself all the time, you'd realize there's so many greater possibilities here that checking the weather and sports scores is just not really something you should waste any time on. It's not, it's not important. Like there's so much better stuff you can do.
- [40:40]
A
Yeah, it's definitely not in, in my view a great way. Like if you say you like researching like historical information about sport. Sure. Like, it's amazing. But it's like I think there's certain use cases that, that, that suited and, and not. And I also think getting to pick the mix of mcps, right. And maybe this stuff will come. But like say you want to do like medical research or whatever. You know, you can create an assistant, add a bunch of MCPs and say I want my, you know, I want like WebMD or like all the medical like MCPS as my, as the things that it'll go and research. And I like, I want to sort of contain it and say, like I want you to use these tools and.
- [41:28]
B
I think even just, yeah, like, even just your own Personas, like you've got Your work Persona where it's like, okay, I'm connected to my work Gmail, my work calendar, my internal work help system knowledge base. I have information in there about the different systems I'm in. And I'm working in that paradigm with that mix of mcps where I can be that person and have the agent do things for me in that context. But then I might want to switch into my side project, side hustle project or whatever it is. And it has a different mix of mcps. Treating it like one universal console just isn't as useful because you don't want it distracted by all the other stuff.
- [42:05]
A
Yeah. So like moving on a little bit. I mean we've kind of covered a bunch of this stuff, but just, just some of the other things that were announced. So they announced this agent builder and I like, I'm not like, I'm not trying to be negative here like for the sake of it, but I just, I struggle and I think a lot of people on X, I'm not the only one. And it is a bit of a bubble. So maybe I'm, I'm misguided here. So they released this visual interface builder for what they call agents. Right. And so this is very similar to what zapier and that N8N and a bunch of, there's a whole bunch of tools that, out there where you can have this like drag and drop sort of journey style visual canvas to build out what they call agents agents. And so these, what it does is you sort of start and then you, you can have like jailbreak, they call it jailbreak guardrail where it's checking inputs. And they gave a few examples. But I would describe this more as like a skill or some sort of like automated task that occasionally uses AI to make a decision. I think that would be a pretty fair assessment.
- [43:25]
B
The second you put a flowchart up, you're like, that's not agency because you're giving it a procedure to follow. You know, like you wouldn't have an employee who has to follow an exact procedure every time and say, oh, they're an agent of their own devices. Like they're, they're out there being a manager, making calls, making decisions. They're not, they're following a strict set procedure with criteria around each part of that thing. And sure they may be able to make a decision in the way that the hungry Jacks worker can decide whether to, you know, give some free fries or something, but they're not actually strategic, they're not actually using intelligence at any point. In this step, and to me, seeing a flowchart is the opposite, like the exact opposite of what I would call an agent. An agent is something that you give a goal to. And yeah, sure, maybe it's seen ways things have been successfully done in the past, but I would argue that that's a layer below agency. The layer below agency is the workers who are following like you say, a skill, a predefined procedure with criteria in it. But the agent itself is deciding when those things happen and with what inputs and how to assess the output outputs of those things. It isn't an agent, isn't the thing that's just following a blind procedure.
- [44:39]
A
Yeah, I think that the confusion for me with this agent builder thing is who is it for? Because any like, so they have the agents SDK, right? Which in theory you can just plug the documentation of that into an AI and like vibe code an agent with that. Right. With their framework, if, if that's what you want to do. And I think if you're a developer, that would be far quicker than some drag and drop builder. And then on the other side of the equation, it's like, okay, well, maybe it's for the business user to describe a process or a task that they want to automate internally using the power of AI. Cool. But then you use it and you're like, this is so advanced that like, you know, it looks pretty and nice, but I played around with it and you know, it's like, which vector store id, how many results to return? What reasoning effort do you like?
- [45:28]
B
It's.
- [45:29]
A
There's a lot to it. Right. And so I can't imagine this empowers people in an organization to go in and like, teach it skills or teach it repetitive skills in their job and then automate those skills, which I would think would be the purpose of something like this. Like, this would be why you would maybe, maybe like build a visual editor. Whereas I look at like N8N and Zapier and those kind of things and I think those solutions are pretty accessible. Like, especially zapier, like, you know, if you spend a bit of time, you can really figure that out and it's, it's quite, you know, quite easy to use. But if I, like, I've got it up on the screen here, it's paused up. Yeah, my whole browser has paused up trying to use it.
- [46:13]
B
Is that them or you?
- [46:14]
A
I don't know. No, I think it's them. Oh my God. It's going to crash my whole browser. So, okay, not a great experiment. Let's not put shade on, here's your, your Zapier example. Right.
- [46:28]
B
Like, I don't think Zapier is trying to claim that Zapier is agency. Zapier is just connecting one tool to another and leveraging the benefit of that. And when they provide their MCP tools, that connection and its tools within it become something that your agent can actually then use. They are just skills in the bow quiver of the AI's agency which can then use. I just don't think the future is going to be building these procedures or if it is, it's going to be demonstrating to an agent. Here is how you do a task of this nature. Remember that. So when it comes up, you know what to do and then as it builds up a whole list of those skills that it's then able to use that in its decision making and it's achieving goals for you rather than calling each one of those things an agent.
- [47:19]
A
Yeah. To me it seems like what it would excel at is if you like visually wanted to just describe a process where you like calling a file search and then you want it to use this tool and you know, it had to do a certain set of steps every time for like maybe like a, a travel agent or travel bot or whatever. But to me, like this feels, I.
- [47:43]
B
Mean, isn't that just, isn't that just programming? Isn't that just like writing code? Like, I don't really see where the AI bit comes in.
- [47:50]
A
No, I, I don't either. And maybe, like, maybe I haven't digested it enough and like I'm wrong here. But I just think if you, if you're someone who can code already or has a fundamental understanding of code, it's far quicker to just vibe code this shit out at this point. And if you, I actually, I don't.
- [48:07]
B
Have my beeper today, so if I.
- [48:08]
A
If I go off on one swearing I'm on standby. But yeah, so I, I think that, and then on the other end, if you're not sophisticated enough in, or not, I shouldn't say sophisticated, but you don't have the skills to code, which is like most people I would imagine, that actually want to do stuff and automate processes in their job. Like, okay, so this isn't, this isn't good for that either. So to me you have that problem of like, who is this for? And I don't, I don't think they know. Like, I, I, I don't get it. It, it reeks of one of those projects where it'll either evolve quickly and they'll. They'll listen to feedback and simplify it, or they'll. Because developers don't want this, like.
- [48:52]
B
Or they'll ditch it and all the people who've taken the time to configure their stuff will lose it. Like, it. It's risky as well for a company to use this now.
- [49:00]
A
Yeah. I just think we're at a stage still where you really want to control, like, you want to control the workflow in code yourself and take ownership of that now for the rest of the episode. Because this thing crashed my browser, I can't show any of the things I wanted to show. So that's a little bit disappointing. And I'm too scared to crash this browser because it may crash the whole podcast recording. But I do want to talk about a few of the other things they released. So they released the GPT5 Pro Pro model, which I can no longer tell you the pricing of, unless I somehow remember it.
- [49:31]
B
You ready with the beeper mic?
- [49:33]
A
Yeah.
- [49:34]
B
It's expensive. It's too expensive. You got onto me immediately and said, chris, you need to add this into Sim Theory. Which I did. And then I looked at the cost and I'm like, hang on a second. This is going to use up, like, with the multiplier, we need to apply so we don't go out of business same day. I'm like, people are going to use up their entire token allocation on their first request. It's like, I think it's like 15 to 20 times more expensive than GPT5 or something like that. Maybe more. It's. It's untenable. Expensive to the point where I was scared to try even one request.
- [50:17]
A
Okay. I've been able to load up in Microsoft Edge because Chrome is actually fully paused up and as I said, too scared to touch it. Now someone said, actually, sorry, that Visual editor of theirs uses like something like four gig of memory. So it's very poorly optimized. A lot of it, like, just to reflect on a lot of it feels very rush. The app stuff to me feels very rush. But yeah. So GPT5 Pro 15 input per million. A hundred and twenty dollars per million. Output. 400k context window.
- [50:53]
B
Yeah. And they allow. I think it's 280,000 if I recall correctly.
- [50:57]
A
I probably don't max out, but. No, you're right. 272 max output.
- [51:01]
B
Yeah. So if you output 272,000 tokens, I mean, like, you're going to cost yourself, what, 25 or something more on a single Request.
- [51:11]
A
Yeah, more, more. It'll be like 35 or 33. Three or something around there. Yeah. It's insane.
- [51:17]
B
What is the output? Is it like where to dig up buried gold? Literally maps. Gold Treasure maps is the only thing you could really be asking it for that would make it worth the price. Yeah.
- [51:27]
A
And like they had that that like AGI benchmark. They're like GPT5 Pro is the most amazing, you know, model ever. But I would sort of argue maybe the benchmark should take into account like affordability. Like, are you really getting that much more intelligence?
- [51:46]
B
It's actually, you know, quite a smart strategy. Make it so expensive no one can verify your claims. So you're like, oh no, it's AGI, but no one can actually afford to try it so they can't tell you you're wrong.
- [51:56]
A
So here, like just, here's the. I'm going to try and actually switch over my. This is just insane. But I'm going to try and switch over my browser at some point so I can put it up on the screen. But yeah. Oh no, that didn't work.
- [52:10]
B
I can see it.
- [52:12]
A
Yeah, it's like my whole screen now. So. Whoops. But yeah, we're technical geniuses, just so you know. Yeah, let me. Okay, so I've got the paused up. The paused up. Yeah, the paused up thing here. Okay, so I've switched over to Edge. Yay. Thank you, Microsoft. So GPT5 Pro, it's reasoning, highest speed, slowest input, 15, 120 output. I look it, I'm sure it's a great model. I know people will drop in the comments, what an amazing model. And I, I hope that this amazing model comes down in price over time and people can actually benefit from that intelligence because I think right now it's just so hard to get any benefit from that with how, how pricey it is. So the, the other one was, was the Real Time Mini model which I honestly deserves an honorable mention. Here I am because it, it crashed my screen. I'm frantically trying to get details of it, but I can't find it, unfortunately. Oh no, I can GPT Real Time Mini. So this is the, you might know, this is like the voice, basically the voice model, like chat GBT voice. Now we've said time and time again on the show like it's just too expensive so you just can't use it for any real purpose. But I think now you can. So GPT Real Time Mini. It's a cost efficient version of GPT Real Time. It's it is insanely fast. It is blazingly fast. I. It defies gravity. Like it. It is so good. It's $0.6 per million input and $2.40 per million output. Very affordable. Very.
- [53:59]
B
Especially in comparison to the Pro. So just to explain to everyone the sort of challenges of having a real time voice model in your product that we have, we've faced in the past, if you think about it, there's quite a few steps in what needs to happen to have a voice model going. Firstly, you need to collect the audio input either as real audio or you can use the browser's text to speech to text ability. Now, it's not that great. It's slow, it puts resources on the user's computer and there's downsides to that. You also obviously have to wait for them to finish speaking in a lot of cases in order to do your inference because you don't know when they're done. The alternative is speech to text where you can use Whisper or another open source or paid model to do that. Then you need to run your inference. Then you've got to run text to speech, send that back to the browser and then play it to the user. The upshot of that is you end up with very high latency, especially if you're using a model that's very heavy and has to consult resources and MCPS and all that stuff. It's not a great experience. Essentially with a real time model, the benefit is that you can actually essentially get very fast responses and voice wise respond to the user very quickly. The downside then becomes that the models are never as good and never as smart. Someone expects that they're going to get the same performance out of the application in terms of its AI reasoning, but also real time voice. Getting both running parallel is difficult, a model like this. What Mike was saying to me this morning is it seems like it's a pretty good model, especially in terms of tool calling, where you may actually see the best of both worlds. Where you can say, okay, I've got the real time voice. The MCP can call off to a smarter agent now if it needs to in order to do that more difficult inference. But you've got that, that real time voice experience while still having the benefit of the intelligence through the tool calling.
- [56:00]
A
I told a lie. So I did scroll down. That's tech tokens. Audio tokens is actually input $10 per million output, $20 per million. So not as exciting. But still, I mean, it's cheaper than I think. Opus.
- [56:16]
B
Well, it's okay. We tried a Real time thing before. And what got us was if you have the sort of open mic experience where it isn't pushed to talk, you're essentially having to always check the audio to see if they said something and they might sit there in silent contemplation for an hour and you're paying that whole time. It's not possible to do it that way. I haven't looked into this model in detail, but perhaps it has a way where you're really only running the inference when there's something worth inferring. And so I think that if they've overcome that, then this is something that even at that price would probably be reasonable to provide.
- [56:51]
A
So I had a little game for us to play which was called VO2, sorry, VO3 versus Sora. So I did want to reflect on this. So they released the Sora 2 API as well. So Sora 2 and then Sora to Pro into the API and Sora to that version of the model is actually like for a video model, pretty economical. So it's 10 cents for either like a TikTok style or, or a landscape. 10 cents per second. So and it goes to a maximum generation of 10 seconds, minimum generation of 4 seconds. So it's like 4, 8 or 10. It will output and if you want to go to Sora 2 Pro, it's $0.30 per second and it, if you want super high res, it'll pop you up to like 50 cents per second. Now 50 cents per second if you're working on like a CGI effect in a Hollywood film. Super cheap. Super cheap. And sort to at 10 cents. If you're just playing around, I would say reasonably affordable to compare to VO3, you're looking at $0.15 per second. So it's like $0.05 more now. It has no watermarks, so it's not like the Sora to Social app, which rip. It's gonna die soon. I bet one week later no one gives it. No, I think, you know, Mark Cuban's got a cameo on there, so he's probably going to hold the fort on that one. But yeah, so sort to. I don't know. I think so here's my theory, right? It's a great model, great at following instructions, great at camera cuts, really good audio, amazing training data and graded instruction. Following is my summary. But if I'm like really wanting to use this model, am I really gonna pick it? And I think what I've seen so far playing around with Sora 2 and Sora Pro 2 Pro testing is VO3 is better. Like, VO3 is just better. It's just way more inaccessible because it's, like, pricey. It's really hard to access. And, you know, it took a while for that price to come down, so people just didn't play around with it that much. But I think in terms of, like, bang for your buck, VO3 is just far superior. And, yeah, it just produces better results. And I think if they had just released Sora 2 as a model, an API, people would have been like, yeah, VO3 is better. So what? Like, that's my opinion.
- [59:28]
B
It was like the distribution that made the difference.
- [59:30]
A
Yeah. And that TikTok style distribution where they focused it on, like, camera cuts and producing something of quality that could be shared with people and shared with people interested in AI to be like, hey, friend, look how far it's coming. And look at this funny, hilarious joke. And it does have a great sense of humor, that model. Like, let's not diminish it. So I think that kind of all. All made sense. But I kind of get the impression maybe the launch strategy to get some hype around it was. Was. Yeah, just that. To build hype. Because if they just pump these APIs out, I would have been on the show being like, yeah, VO3 is better. Oh, my God, they're behind. That's the truth. And so let me play. I made you two examples. We were working late one night, and so here. And my camera's gone out. So what a great show. But here is good night, Chris, and I'm not going to tell you which one.
- [60:22]
B
Mike sends me a loving good night every night.
- [60:25]
A
I just realized my entire. I don't even. Like, I can't drag and drop anymore. So that. That's over. My camera's gone out.
- [60:33]
B
It's done. That's the end of the podcast, everyone. We quit.
- [60:36]
A
Yeah, Honestly, I think that's gonna have to be the show because I. Everything's broken and your audio is terrible. It's. Yeah, it's been a tough one, but yeah. So anyway, rather than show, because I can't, you're just gonna say goodnight to me. Yeah. Good night. The VO3 is just so, so much better. I really wish I can play the audio, but it's not worth it.
- [61:00]
B
I can see you again.
- [61:01]
A
That's good. I'm very blurry though, Now. So anyway, VO3 is just so much more impressive. If I was, like, actually working on serious video work or trying to put these models into, like a ser. Moment, there is no way I would pick Sora at all. Even in the Pro Elite plus version, I just. I. You know, it just doesn't make any. Any sense. And so, yeah, that concludes the episode. Now, we. We. We are aware of Gemini 2.5 computer use. Given our technical difficulties. I think we'll. We'll cover that next week and hopefully show you that working in SIM link, because I think that's a good target. Yeah, it looks real, real good. Now, finally, a reminder. A reminder that, Chris, we're like, we're. We've got two. Two albums out. We were meant to have three albums out. So now on Spotify, before we go today, you can get all of our DISS tracks. All of our tracks. There's two albums. There's. One is called Average Tracks from the Show. Average Tracks from the Show. The other is called AI Diss Track Collection. And we'll try and keep these somewhat updated.
- [62:15]
B
And so what happened with the musical? That we're just intimidated by how good it was?
- [62:19]
A
No, the musical had a few technical difficulties. And the entire musical, for those that liked it, the two people out there. Yeah, that will. That will actually be available hopefully this week. But on the average Tracks from the show, there's songs like Billy's in the bank, like, let's just play a little bit, shall we? Just to remind people. So a few hilarious things about this. I actually had these tracks mastered to be able to put them on Spotify. So these are like leveled and mastered. And don't ask how much it costs. We'll never reveal it. There's also this one, which I think deserves another listen to. This is Born in the usa, but it's all about the history of AI. So if you listen to it, you'll learn about the founding of AI.
- [63:20]
B
Play it to your kids. It's educational.
- [63:22]
A
Yeah. And then of course, you've got the GPT5. Sad song, spinning another life.
- [63:29]
B
GPT5. You're supposed to be so smart, but you ripped our work.
- [63:37]
A
And of course, how could we forget? Very recent track, actually. And then, Chris, I actually think only because you forced me to put this one on there, this album reaches a weak point towards the end, all the ones I made. But yeah, so there's there's some, like, honestly, some of the best DISS tracks in the world. I'm that constitutional AI trained up right and proper. While GPT's out here playing, I'm a showstopper. You might have more parameters, that's your claim to fame.
- [64:18]
B
But when it comes to balls and.
- [64:19]
A
Flow, you're looking mighty tame. C L A U D E. Yeah, that's me writing for. So now your family can hate you on road.
- [64:26]
B
I'd love to think that maybe some. Some of the, like, lesser workers in the anthropic office put these on to, like, motivate themselves and stuff.
- [64:33]
A
Yeah. Yeah, I. I would. It would be nice to know that Dario has seen my pendant necklace as well. Leave a comment below Dario, if you listen to the show. All right, that'll do us this week. It's. It's. We've had a pretty shitty week. I'm gonna not gonna lie. Some personal stuff's happened to us, so we did want to bring you an episode and try and, like, unfray our brains from what we've been up to. So hope you bed with us. And listen. Sorry about Chris's audio quality. I don't. I don't know. It could be fine. It could be fine. Let's hope it's fine. All right, we will see you next week. Go on to Spotify, wherever you get your music, and. And. And listen to our songs. Maybe we'll beat Taylor Swift's album. Who knows? We'll see you next week. Goodbye, Sam.