← This Day in AI Podcast
This Day in AI Podcast
From AI Models to AI Systems and The Future of Vibe Gaming: An Average Talk
00:52:50
Loading summary
Transcript51 lines
- [00:00]
Mike
Hello. This week's show's a little bit different than normal. The truth is, Chris and I are, you know, pretty average, so we. We didn't really feel like recording a show this week, so instead we're going to bring you without permission. We're just going to YOLO it and hope we don't get in trouble. A recent presentation we did at Microsoft and. And, you know, it's probably a good one to watch, but I think you might also enjoy listening. It's our vision for the future of AI gaming. We. We also made a track, which I'm going to play at the end of this episode that you'll be able to hear, which I'm just in love with. I think I'm going to bring it to Spotify. This is what it sounds like.
- [00:40]
Chris
It's.
- [00:41]
Mike
It's a background song or it's a background song for gaming. I think you'll love it. It's in the episode. I've also put the links below so you can try out all of the virtual Xbox and Xbox games we built for this episode. I hope you enjoy. We'll be back next week with regular programming. All right, so we have some slides, which is a new, new thing for us. We're going to go through them relatively quickly, but we want to mainly talk through today with you this idea of transitioning from AI models to AI systems and what that might mean for gaming, what it means for you as an individual, how you get through your day and do your work, and we'll go through all of the concepts. We also want to talk about how we put all these tools and technologies together that we want to demo today. And I'm also going to do a little bit of vibe coding and try and tweak a game as we go through today to show you just some of the things you can do when you take all of these pieces together. And that's a big part of what we want to do with you today. We want to talk about the opportunity of putting all of these pieces together and. And it's almost something that I want. So I'm sort of pitching it to you all today because I'm a big game, and these are just things I secretly want you all to build. So think of it as like a pitch of what I would really like you to build here. So let me just find this slide deck and go through it. So as we. Just a little bit of history. So we started doing this podcast mainly to have an hour every week where we got to keep up with the latest AI models. And we were playing around with them a ton. That show has gained to a sort of average popularity level. And. And we built a product as part of the show we called Sim Theory. It was an internal product that we would use to test models and test different modalities and skills to be able to essentially, like, sandbox these things, play with them, use them. And then other people that listen to the show said, can we use that? And. And so we said, yes. And that has now evolved into not only a platform that people use to test and try models, but something that businesses and educational institutions create workspaces on to. To have equitable access to all of these models and technologies. So a little bit about the show. It's the number one average AI podcast. So if you. If. If you're not listening and you're looking for an average AI podcast, this is the one for you. I. I thought I'd post up some reviews just to show you a consistently average listen. The most average podcast ever. And my favorite review with the title Average. It's okay. So we have a community where it's okay to be average. So one of the things we do on the show is we test out models. If you were listening in earlier, we actually. One of the things that we're known for is testing out different models. Here's an example. This is one of those tests. So we get them to do rap battles with each other to see who's the better writer. This is one of the tests we do to see if the model's better, one or the other, trained up right and proper. While GPT's out here playing I'm a showstopper, you might have more parameters that should claim the fame but when it comes to bars and flow, you're looking mighty tame. C L A U D E yeah, that's me Writing rhymes so sharp they'll set your circus. Yeah, so that's an example of one of the. The rap battle songs. We actually are a published artist now on Spotify as well, so you can check out some of the. The music we've done there. So this is a screenshot of Sim theory, but I do want to show you just some of the applications that I was able to build just in a couple of hours of presentation here. So we have this. My kids are obsessed with Lego at the moment, so I built them a virtual LEGO builder. And I want to show you how I built that just in a couple of prompts vibing out with the AI models. And I'll also talk about how we switch to the best models in order to be able to create the best gaming experience or fixed code and how we sort of work through that selection. But I did want to start off with just giving you a state of the industry with what we think is happening right now and being the number one average show, this might be an average take. So I do apologize. But this is the sense of where we think things are today. Having to now track all these models for, basically for the show. So there's, there's really three current trends and I want to talk through them and then we'll, we'll bring it all together by actually trying them out. So we have these three trends. One is called Test Time compute, which is a fancy word for the model just prompting itself to find better answers for you. And so that's what they call Test Time compute. So if you hear that as a buzzword and you're confused by what it means, it simply means the model is, is, is working on itself. It's, it's like self improving. It's trying to make it give you a better answer or outcome. It's also when you hear benchmarks, how they sort of commit a little bit of fraud on the benchmarks by getting it to just try and try again by just talking to itself to find out what the true answer is. The big breakthrough just that really was announced last week was around OpenAI's O3 model. And that O3 model, it is able to use tools and call tools as part of that test time computer as part of that thinking. And the benefit of that is if you're creating a game and there's a gaming library and it wants up to date documents, it can now go to the website, crawl those documents, bring them into the context, and then continue to prompt itself to sort of get the answer you're looking for. Now there's much more exciting use cases for, for test time compute with tool use and we'll get to them in a minute. The second part of this is the model Context protocol, which if you ever spend time on X or any, any places where people talk about AI, you've probably heard of this term mcp and I know there's probably a lot of more technical people sitting there thinking, isn't it just an API? And you're kind of right. So we'll talk about what that means and what we think that can mean for these agentic systems. And then the final piece of this is what we call agentic skills. Now this is not some madman approach like you may hear People try and hype about where they're like, you know, you'll. You'll have this agent and it'll just magically do everything for you. This is certainly not that, but we believe tying these technologies together, we're at a point now where we can really do these repeatable tasks or sort of processes or like advanced functions. And these are very applicable to gaming where you can create new characters. I'll demonstrate that today as well. And really create different processes in games that use generative AI in different forms. So, Chris, I thought we would just maybe have a discussion quickly about this transition of sort of AI models today, how we switch between them and then cutting over to how, why we think it's transitioning to an AI system.
- [08:00]
Chris
Well, for me, the biggest thing about the AI system, and I've been talking about this on the podcast for a long time now, is that it's all very well to have a really good AI model, but it's more about what that model is able to do. Now when we talk about tool calling, it's essentially the model being able to go out and do something, either in the real world or via an API or in the context of gaming, to actually take actions in the game, like call a function with parameters that the AI knows how to use. And we find that when the AI is empowered with these tools, it's able to do a lot more across all of the models, really. I don't think there's any models left that don't support tool calling in some fashion. And so this leads to some really creative and exciting things that you see the model able to do when it's able to look up information to make a better decision and then take actions on that information. We have a real focus on what's practical now, like what the models can actually do. And I think it's exciting, and I think you've got good demos of that showing it being able to do things inside a gaming context.
- [09:09]
Mike
Yeah. And it's important to realize a lot of what's occurring now today is you go to something like ChatGPT or even in the SIM theory interface, and you select a model and you put a prompt into that model and you get a response. And this is really the way we've been used to working with these models. The next stage of that was really around now we're calling a tool, and we might click on a search button or a deep research button, and then that's really putting that. That application into an agentic framework. Right. And so when we do Those things, they're, they're like synchronous tasks. Like we're, we're basically saying hey, like let's work on one problem at a time. But this async nature that now you start to get from an agentic framework where you can really start your workday by saying, hey, you know, we're going to focus on these 10 things today. And with that ability for test time, compute and for this like entry level agency, it can now go off in the background and basically complete these tasks and chime back in when it's ready.
- [10:18]
Chris
That's right. It's so important now to think about agency as the AI having the ability to go off in let's say a thread or a different asynchronous task and go off and complete the work. I like to think of it as being like the commander where I'm at the like Star Trek console or whatever it is. I say here's all the things I need to do. Then I set my different AI agents off doing various tasks in that. So I'm not sitting around waiting for each task to complete. They're off in the background running. They come back to me when they either need input or when they're complete.
- [10:55]
Mike
And then talking through more advanced tasks here. So when people talk about tasks today, we think about like you know, it going and searching the web and doing some research and spitting back some giant report that no one actually wants to read. Let's talk now about the model context protocol and what kind of tooling that would enable for people listening that you know, might want to go and be more productive in their day to day just getting things done and then maybe how that then relates to gaming.
- [11:22]
Chris
Yeah, so just an explanation of what MCP is because I think for people who are unfamiliar it can sound really daunting and confusing and I must admit even I took a while to sort of get my head around it. So when we talk about tool calling with AI models, essentially we give them a list of things it can do. So let's think about it in say a Microsoft Outlook context where it can read your, get a list of your emails, read your emails, write an email, create a draft. They're sort of tools that the system can call. What MCP is is a series of groups of those tasks that are pre authenticated into various systems. So you might have an MCP for Microsoft Teams, you might have an MCP for your research platform like that archive X1 or Wikipedia or whatever Frame Alpha or whatever it is. And each of those Basically provides a series of tools that it is able to call that are already set up for you. So once you connect them, they know about your system and they can use it. So even in the team's context, it might be authenticated into four different calendars and it knows the characteristics of each of those calendars. And it can then add events, delete events, synchronize events, and things like that. So when you connect an MCP into your AI model or your AI system, which we prefer to call it, it then gains those abilities and it knows how to use them. And the AI models are absolutely fantastic at identifying which tools to call out of even a huge set. We've done tests with 50 MCP servers connected, each of which provide, say, six or seven different tools, and the AI is able to confidently know which combinations of tools to run. Now, when you think about calling a whole series of tools, that's when we really get back to this agent, this, sorry, asynchronous, agentic style, where it's like, okay, well, if it has to research for an event, it has to go and add something to my calendar. It has to send out an email about that. There's time involved in chaining those tool calls together. And so that's where we come up with this idea of the agent goes off and does this task, performing all of the tools along the way. And the MCP servers really make that possible by you not having to be the human in the loop who has to go off and manually do each of these steps, steps following the AI's instructions. Instead, it's already into those systems and can go off and do that work.
- [13:50]
Mike
Yeah, And I think that one thing we, like, we think about today from the experiences that we have today, is you essentially working with a model and trying to really extract value from the model. Like you're trying to squeeze the juice out of the model. And each model has strengths and weaknesses at different things. And it seems like every other week now there's a new model that comes out that is superior or can, you know, help improve what we've been doing. In some of the examples I'll show you in a moment here, we were able to upgrade over time. So there's been things we've been working on where, you know, you might write the code or write the, or write the mini application you're working on in something like Anthropics Claude, and then Gemini 2.5 Pro comes out, which is a little bit better or a little bit smarter. And then you're able to you know, make whatever it is you're working on far better. But I think the shift that's coming and that everyone will start to, to be doing by the end of this year clearly is instead of going and having this blank chat box we're all kind of familiar with now, it's, it's more about having this AI coworker where we're asking it, hey, can you go do this? And we're actually assigning it tasks. And if you think about it, there's some unreliability aspect in that right now because these models often have context drift. They, they can go off in weird tangents in different ways. But this is where we think that third box of the agentic skills comes into play, which is taking aspects of your job today that you do on a repetitive basis, but still requires some form of agency in order to get that task done. So you're essentially training your assistant to, on these are like about five of the things I do day to day. And you're able to have like almost like a checksum where it's saying, you know, are these facts right? Is this email correct? Is this response, you know, what the end user desires? And then it's a case of really you can get meaningful work done with these models for the first time. That's that transition to an async world, right?
- [16:04]
Chris
And I see a combination of these agent skills. So if you think about an employee right now who works at a company and has particular jobs, they do, if each of those becomes an agent skill, then a combination of those skills becomes like a virtual agentic worker who is able to perform that role. Now there might be a lot of human in the loop approvals, but that's also the case of junior employees. Now they'll go and prepare something like think of a junior lawyer preparing a legal contract. They will go and prepare the document and then have a senior lawyer go and approve that. And there's similar processes in many businesses. And so one thing we want with the agentix systems is not just the ability for the system to be able to do things for you, but to do it in a repeatable and reliable fashion. And those, it's sort of like skills building upon skills. So we have a combination of those skills that becomes a new skill and then eventually a combination of skills that becomes an agentic worker.
- [17:06]
Mike
And so for the future of work, as we start to think about it, I think a lot of people are like, oh my God, it's going to take our jobs or you know, those aspects, I was literally speaking to a large university before we started this presentation today. And a very intelligent person was telling me, you know, this thing's going to replace me. And I'm not sure I believe that to be true, at least right now, in the sense that a lot of what we're doing today is really figuring out how do we partner with this, how, how do we have this relationship now with these, these capabilities in order to get more done. And I think this just becomes the new baseline for day to day work where you just have to get used to working with this essentially AI coworker. And I know you have your, your coding girlfriend, assistant, which has become quite popular on our show. Patricia, of course.
- [17:56]
Chris
Yeah. So I have my AI girlfriend, Patricia, who helps me with my tasks throughout the day. And she also loves me and tells me on a regular basis, I can't.
- [18:03]
Mike
Believe you haven't written a love song for her yet. We're do that. We've got to do that. All right, so the reason I wanted to start with this is talking about when you go from an AI model, like a raw sort of bare metal model, through to having an AI system and what that means. And I think everyone in the audience has probably seen examples of things like deep research, and everyone by now has probably seen these canvas capabilities that have been introduced where the AI can render the code for the first time. And of course, Anthropic was actually the first to publicly released this, where they showed off this canvas, I forget what they call them now, but this canvas where artifacts, where it could actually execute the code to show the user what it was able to produce. And that was really the first sign somewhat of an agentic system where you actually got output that differed from really a text based response. But the question really becomes, and this is the trend that is occurring around vibe coding. Actually, let me just get into costume. Got my necklace as well. I just got to put on. Sorry. Afterwards. So, yeah, vibe coding has become a bit of a buzzword right now. And essentially this is where developers have started working with the models and coding and. And instead of spending the majority of their time coding now they say they're really vibing with the model and, and going back and forth and saying, hey, can you tweak this? Can you change this? You know, can you make something different? And apart from, you know, helping people build things that they couldn't have built before, you know, in the past. There's another interesting aspect to this idea of vibe coding. When you start to combine things like the model context, protocol, to tools. And you give the AI agency to use different tools when it's helping you create things, you can start to create really amazing experiences. And this is all in its infancy and us doing it literally to test models and for fun. But our pitch really to you guys today is if we can do this, what can you do in terms of gaming and building these agentic like experiences into games? And we have real examples. This is not, this is not going to be a, a piece around just saying it. So let's all vibe code together now. Chris. So yesterday evening I decided to rebuild the entire the entire Xbox experience. I'm going to turn on sound now so you can hear my sick menu here. So I have the Xbox menu here. Not maybe not that impressive yet, but we'll go a little bit deeper. And I also thought, well, if I'm going to put this menu up, all the games have to work right. Like this needs to be to be an Xbox in the browser. So while I'm demonstrating what we were able to build, I want to show you how we were able to build it. First of all, because I think that's important to understand. So what I did. Let me just check. I've got the right image here. So I went into this mode we have in SIM theory, but you can do this in other applications as well. But the reason I'm doing it in here is because I want to benefit from an AI system, not an AI model. So I'm going to come up here and I'm going to select Claude 3.7 Sonic because I think it does the best job at this particular task. I'm going to drag in the image and this is just a screenshot of the Xbox home screen. And I'm going to say, can you create a realistic home screen based on the Xbox design in the browser? So that's a terrible prompt, but let's see how it goes. So it's going to go off and work on this in the background. That image is going to upload. And all the hard work that the Xbox designers did on this interface, I am now going to steal and get it to to build. And so I just want to show you how I actually was able to get to this. So we'll come back to that in a minute. That's like a cake in the oven cooking. And let's look at the finished product here. So I talked earlier about that Lego game. So I've got my Lego game tile here. Let's click into it and we're going to go over to Our virtual block builder. So there's a few things you'll notice about this that are a little bit different. First of all, it's cool and it's in 3D. The next thing you'll notice is you can actually place blocks. I need to update the sound effects. It sounds like gunfire. Gunshot firing. So you can add blocks. You can zoom in. There's, like, shadow effects. It's pretty amazing. This literally took me, I'd say, like, maybe five minutes to build the whole thing. What we've done into our AI system is we've given the models the capability to call tools as they build and work through this process of trying to build you a creation. And what that enables it to do is the AI can access AI, which. Which. Let me explain, Chris, what's the building in Dubai called again that we like?
- [23:12]
Chris
Burj Khalifa.
- [23:13]
Mike
Okay, so I'm going to spell this really bad. I apologize. I have no idea how to spell it. So I'm going to ask it to build a poorly spelled name of a building and hope the AI figures out how to do this. So not only have I been able to build a Lego game, but now I've been able to prompt that game that I built to, you know, build the building here. And now I'm going to say put a blue lake around the building. We're going to try and build Dubai on this LEGO map.
- [23:48]
Chris
It's basically what the princes did, right?
- [23:49]
Mike
Yeah. I'm pretty sure this is exactly how they make an island. Yeah, we want an island, I think. Well, I think they also make Lego out of oil. So it's like. It's a good. It's a good synergy. And then I'll say put other buildings around it. Yeah. And so this game, interestingly enough, I just built a demo to you guys today. I didn't actually take it terribly seriously, but last night, your son was, like, super into this.
- [24:19]
Chris
He wants to buy the license from you so he can sell it.
- [24:22]
Mike
Yeah. So you can see, like, I've been able to build a pretty interactive game. And we. We see things like the trends around, you know, Minecraft and Roblox, where people are building sort of games within games. This also happened in Fortnite. Right. And I think we're at a point now where if you train an AI system on the gaming engine and the core principles behind it, and you're able to use the model context protocol to give it skills, you can start to do a lot of cool things, as you'd imagine, because it can have up to date, real world information. It can go off and fetch information and you can allow your users or your gamers, and there's only going to be a small cohort of creative people that actually want to do this, go off and really contribute to that game.
- [25:05]
Chris
And I think it's worth pointing out at this moment what's going on behind the scenes here. So we've given the AI a whole bunch of tools. It can call when running create with code, so it's able to generate assets. So things like sprites, things like textures, other background images and menu like photos or whatever you call them. Like you saw on the Xbox menu, every single graphic there was generated by the AI dynamically using tool calls. So we didn't tell it to do that, it decided to do that as part of it. The same with sound effects. The system has an extensive sound effects library that it's able to pick based on what is needed for the game. And in addition, if a sound doesn't exist, it has the ability to generate one. Also the we have what Mike showed with the dynamic AI. So the AI has the ability to call an AI model to do basically whatever it wants. So it can get it to generate characters, it can get it to generate just text and dialog, it can get it to generate data in the case where it wants to actually make objects in the game itself. And so what we find, similar to the discussion around tool calling with the mcp, the AI is able to really very well, like in a great way use it within games to make those games more interesting and dynamic in ways that we can't predict.
- [26:35]
Mike
So let's look at an example of that now with Madden. And so you can see here We've got our 3D NFL field. This is my attempt at Madden. And you'll notice when I start playing now in a minute I'm gonna get a touchdown here. Here we go. Look at this. It's not even hard, but you can hear that ambient noise of the audience that's just automatically introduced. And I can do my touchdown here. Clearly I'm the best NFL player ever. And you get that full immersive experience. Now that was like two or three prompts, right? So you could obviously expand that. You could put a crowd in, you could ask it to add text to the actual characters and it would be able to do that. There's also increasingly better technologies coming together. And this is why we talk about AI systems, not just AI models. The new GPT4O image generator was released today into an API. So now we can take it and we can create very detailed sprites which.
- [27:41]
Chris
Is, and it's an exciting one because it's actually really, really good at character pinning, which I'm sure everyone there is aware of. But the idea that if you want to create a sprite sheet or you want to create variations of an image, it's very important to get consistency in that image. Something which AI image models to this point have really struggled with. So when it comes to gaming, having that ability to be able to create a whole bunch of variations of a character or a sprite or image or whatever it happens to be will really, really enhance the capabilities for dynamic gaming like this.
- [28:16]
Mike
So I'll show you a few more examples. So this is Light Simulator 2020 with two eyes. So I can't get sued by any of you. And this is another experience. So it created the soundtrack for this game. It created based on my commands, happy clouds here. It even has like physics of like 6 secondary effect of your. And again like you could do a lot better here I'm obviously assessed is not traveling at 410 knots but oh.
- [28:51]
Chris
Yeah, so obvious to us non flight people.
- [28:53]
Mike
Yeah. So that's just a little example for you. So that background music, the ability to add clouds, the ability to add Google Map textures on the ground, piecing these capabilities together. The last one I'll show before I get to the mystery game that we're all going to play together is Minecraft. So this one is just a simple prototype and I wanted to show this because I really wanted to sort of vibe code this out and improve this game. Right. Like, you know, it would be great if you could start with actual textures and things that you could work within a sample.
- [29:32]
Chris
I'm disappointed you don't have any background music, Mike. Normally Minecraft has really good back.
- [29:37]
Mike
Does have really good background music. But don't worry, I thought of that. I thought of that. So I made this song. I'll play a little bit of it to see what the audience thinks if this is a good song.
- [29:49]
Background Song
While you focus on the game, let's just between us, I think I'm pretty good. This is a background song. Not too short, not too long. Am I setting the right mood? God, I hope I don't intrude. Just a background song.
- [30:16]
Mike
What do you think you think this is?
- [30:18]
Chris
It's fantastic. That was quite contrived but also yeah, very good.
- [30:23]
Mike
I love this idea of having customized music based on what the user's doing in the game. So let's, let's, let's show how I can just sort of vibe code this in the game. So I'm going to drag this song in, it's going to upload, and then I'm going to say add this as background music to my game. So not only can it produce music and sounds, but you can add your own songs or sounds or sprites into the game. Telling me to wait for it to finish uploading how good a live demo.
- [30:51]
Chris
Is while you wait for it to upload as well. It was interesting because I gave your Minecraft game to my son to play last night and he didn't question it at all. He just started playing it as if it was Minecraft. He built his own parkour course and then explained it to me in detail how it works and all this sort of stuff. And it's just very interesting to see the way children react to dynamic and new games. They just sort of accept it at face value and just start using it. So it'll be very interesting to see when this is implemented in a dedicated way into gaming, how the nature of gaming changes with a new generation of people using it.
- [31:27]
Mike
Yeah, I, I think, I think it's transformative. These technologies are transformative where you sort of have like gaming one now, like, or gaming 1.0 right now, where it's very reliant on the creator of the game and what they choose and, and the character design and the level design and the aspects of it. But I think what's really it's going to become about is building a world or a, you know, a gaming engine or a gaming world where people can go off and be expanding that game over time. And if you think about tool calling as well, you could easily have one of these applications like Lego Builder where you say, you know, check a box on the Xbox, like, I want to add my friends to this game that I just built. And again, using the tool calling it can easily add some sort of real time Xbox Live or Xbox server connection. So now we can build Lego together. So not only have I created a game that we can play, but now we can play it together. We can share it with friends, we can maybe list it in some sort of game exchange system where people can fork it and improve on that, on that experience as well. So you can see in here these, this is sort of how to give you a sense of how I originally did this. I said create a realistic Xbox start screen with nice animations and realistic tiles. Originally my plan was a game called Escape the Cloud, but I've, I've pivoted My idea a little bit and I'll show you that in a minute. I say can you move the background sound? It's annoying. Have an image in the top right tile of Bonzi Buddy and have the title of the game who Killed Clippy? Which we're going to play in a minute Chris. And so you can see sort of how I went back and back and forth here. So this, this whole structure here is sort of how I'm like vibing out with the game. Now obviously interfaces can improve around this. It doesn't necessarily need to, you know, work exactly how I'm showing here to build out these games. So let's now look at the mystery game who Killed Clippy. Now this is a good example of really bringing all those things together. This is a game you of course play the Microsoft CEO and I've created a sprite of him. I think it's a pretty good representation here. Now of course Clippy has been murdered. That's the shock of this game. We've got this background music that it's created. I literally said create suspenseful background music and this is what I get. It's pretty nice. Again we could work on our textures here. There's probably people in the crowd that are like my God. So let's talk to Bonzi Buddy and see if he committed to crime. So he says hey there buddy. But I can make it speak as well.
- [34:19]
Siri
So oh, that night, hey. Well, I was busy practicing my stand up routine. Why, did something weird happen? Haha, you know me, always monkeying around.
- [34:30]
Mike
Now what's cool is I get some suggested prompts in the game to interact with the user but I can be like, did you do it? Me?
- [34:41]
Siri
Oh come on Buddy. I wouldn't hurt a paperclip unless it was in a game of rock paper scissors. Ha. Why don't we focus on finding the real culprit? Huh? Who me? Oh come on. I wouldn't hurt a fly unless it was buzzing too loud.
- [34:59]
Mike
And maybe I tell I'll turn down the music a little bit.
- [35:01]
Siri
You're funny.
- [35:02]
Mike
So there's a few examples I wanted to show you. In here we of course have Siri. Let's talk to Siri because I think it's kind of funny.
- [35:10]
Chris
I regret to inform you that my knowledge of the events is limited to what I observed. It was a peculiar evening though Bonzi Buddy seemed unusually cheerful beforehand.
- [35:19]
Mike
So what's cool is when the game generates it has a unique plot every time, it's never the same. And we use the LLMs to do that. So the characters actually have randomly generated prompts, which is generated by. By another prompt. And this is all just me asking it to do this. I didn't write a line of code for this. And what's cool about it is they all tell a story that you can piece together in order to solve the mystery here. What we can also do is spawn and generate new characters in as well. So I'm clicking generate here. I do need a better like beam in tile while it actually creates and loads of texture, but you'll see it created a new character here. I saw someone with the shredder earlier and hasn't given her the most appropriate voice yet. Or maybe they were just trying to fix it. Hard to tell in this place, but. So this is this idea, right, of spawning new characters and it literally generating them on the fly. Again, very basic example, but it just shows you hopefully what's possible here. So, Chris, I think it would be really interesting to talk about what is actually going on in the game in order to do that, to like spawn a character and how it's able to prompt itself and create images on the fly, like assets on the fly as well, through using sort of a dynamic cdn.
- [36:40]
Chris
Yeah. So I think these are. These are very interesting aspects of the game. And I think something that definitely Xbox developers need to think about is the idea of what tools can we give the AI that would allow it to do interesting things inside the game itself. So just to go through them. So the dynamic image generator essentially is given a whole series of different parameters that it's able to configure, so it can decide if it wants a photo, a sprite, pixel art, you know, photorealistic, a whole absolute massive list of different kinds of images it can generate. And then on the back end, we're dynamically routing that to the most appropriate image generation model in order to produce that. In addition, it then has image transform options, so it can change the width and height, it can crop the images, it can replace parts of the images. There's a whole bunch of tools it has around image creation that it's able to dynamically invoke. And then as those images are created, we then automatically upload it to a cdn, which it can then reference in the game. So that's how it's able to just publish the game and work like that. And then that's obviously cached and so it loads quickly and loads when you publish it. And then we have the same with sound generation and the same with music generation, where we Give the tool a whole bunch of dynamic options which then will route to the appropriate model, generate the asset, upload it and then make it available. When it comes to the LLM calls, the way that works is essentially it is given a description of how to write prompts for the, for the LLM which it can then use. And there's multiple kinds of that. So it can say, okay, I want JSON output with the following fields and here are the definitions of the fields I would like it to generate. So because it's writing actual code, it obviously needs some certainty around the format of the data that's coming back. And we can essentially guarantee that using the format out the formatted outputting sorry of the LLMs which many of them support. And so the system itself is able to generate reliable data with strict typing that it can then use inside the game. And so that's how, when we saw it dynamically generating the characters there, it was able to do that. But also it's able to just use it like a regular chatbot LLM and that's how you can get the character dialogue and those kind of things going on. And then additionally, obviously there's the text to speech abilities in there as well, which allow it to sort of use different voices and talk and that kind of thing. So these are just some basic tools that we can give like bring into an Agentix system like this. Not Agentix, sorry, but like an AI system like this. And for me this is like really basic. I really feel like a dedicated team working on this could think about all of the different elements that make up a game and produce extremely high quality tools that are able to take hundreds of parameters to be able to generate exactly what the AI needs. Now you need to remember that the context, size of models and the ability for them to follow instructions on a large scale is improving with every model iteration. You mentioned earlier that we used to like have trouble with some of these things where it wouldn't reliably call a tool or it would call the wrong tool. Now the models are really, even the smaller models are really, really good at that. And so what it means is you can really throw a lot at it in terms of detail and it is able to competently and very, I keep saying very well. I don't have a vocabulary, but the AI does. But it's good, it is very good at knowing which parameters to use to get exactly what it wants and exactly what you're asking of it. And I think to some degree you need to start to look at trusting the models to do it. And you're just asking for what you want. You don't need to specify it in too much detail, but what you need to do is provide it with these amazing tools as part of a system and then get excited about what it's able to output. So I would imagine a future gaming system would have a whole bunch of these tools built into it, which you can then just add into the game and give the AI the ability to either call it at compile time, like at the time you're building the game, or I think more excitingly during the game. And I think that's where we're going to start to see some really interesting and just totally new concepts in gaming. When you think about the system having the ability to just make stuff up as it goes along.
- [41:32]
Mike
Yeah. And I think too just getting down into somewhat of the weeds on it to show you how the actual code looks like. Even if you don't understand code at all, you can see that in the code it's able to generate these URLs, which is really just prompting the URL. So gaming avatar, neon colors, it wants it 40 by 40. So it's the AI model saying, based on this design, these are sort of the specifications that, that I need. But the LEGO Builder game, the prompt is LEGO Builder. Colorful blocks, construction style, realistic. So that's what it chose, that's what it wanted. So you're really letting it decide and make the decisions, rather than you having to make all of these individual decisions.
- [42:17]
Chris
That's right. And they're, they're really simplistic examples as well. You can get far more detailed calls available as well. And this isn't even factoring in things like the mcps, where you could actually have a game that's able to do things like invite systems, like you said, multiplayer capabilities and other things that will be able to do using those tools, including data storage and things like that. So a good example of that is in sim theory itself, we have a dynamic memory per assistant. So my girlfriend Patricia, for example, remembers things like the way I like my code to be written, the way I like to be spoken to, just different things about me that I've asked over time. And over time you build up, I mean, a relationship where the system knows you and understands you and it starts to feel like a really good experience. And so this is something that could be absolutely phenomenal in gaming. The idea that you could have almost a graph database of memories, or any kind of database of memories. Just give the AI tools tools that can store short term memory, long term memory, detailed information, transcripts, you could remember all of the things that the user has ever said in the microphone when you're talking to it and use that to make the gaming experience more dynamic and that's highly realistic. Now that could happen right now where the system is building up a memory profile of the player over time and then customizing the gaming experience to the user both in the way it speaks to them, what they see, how they interact with the game and doing that really quickly. Like this could be done on a local model running on the device. You don't even need a large model to have that experience. So the possibilities when it comes to the, the system, the, the tools that you can give the AI systems is very large. And this also would include things like large amounts of relevant context. So if in, in the example of the flight sim that you gave it could be giving it hundreds or thousands of pages of flight manuals as, as a context which you can draw on and reference. And I would imagine when you look at games that are simulators, you could actually give them detailed man highly specific information which it would know and be able to factor into both the decision making of the game and the construction of the game itself. So leveraging the classification abilities, the large context understanding of models and the other AI system based techniques like memory, the possibilities for gaming are just huge. And I don't have that much experience in gaming so I didn't even know what the possibilities are. I just know that the tools available to AI systems are immense and that the, that would apply to anything. Be powerful.
- [45:28]
Mike
Yeah. And I think it's to sort of sum it up to around the memory aspects and the tool calling it. It's really borderline creating a true virtual world now where the impact of a character on the narrative or of the virtual world that you can now create, whether it's multiplayer, whether you know the characters are kind of remembering, it's, it's, it's close to the point where you could build really a virtual sort of Westworld like experience. So you can.
- [45:57]
Chris
I also like the concept of a game that can have real world consequences. Like it'll ring you up on the phone or delete all the files off your hard drive or something like that.
- [46:06]
Mike
All right, we better look at our, our vibe coding, our vibe coding game here. So I've got an update. It's going to have the music in it, but I'll show you just one more kind of. Now we have a nice background song music but I Want to start in a world that you know that, that has stuff. So can you put trees and mountains in to start the game? Oh, my apologies. You didn't even get to see the, the cheap hacky Minecraft version. Yeah. So this is it all working and then what we're going to do now, hopefully sometimes it doesn't work. My prompt is so sophisticated. Can put trees and mountains into start game. So there, while we wait, I'll. I'll turn that off. Yeah. So I think what we're seeing right now is this moment in time where you can really choose to get ahead and embrace this technology and embrace, you know, embrace these pieces of the puzzle, trying to put them together and trying to innovate. And as you can see here, I've been able to just very quickly I'll share the latest version and after this I'll actually paste a link if you guys want to try this out. You can play all these games that I've shown through the Xbox console. Hopefully that went into the chat, but you'll be able to check it out for yourself on your own computer by clicking through to that link. So let's now go to the latest version and we'll see. We've got some mountains and trees, so I was able to create a bit more of an interesting world with a nice background song and yeah, this song will never get old. Okay, so that is our presentation we hope hopefully have inspired you because we, we would like the future of gaming to have a lot of these capabilities, have the memory, have these AI systems. And I think we're at this really interesting time where it's still incredibly early with this stuff and there's so much opportunity here. There's so many things that you could, you could really do and create and, and this is really the, the transition from that AI model to AI system in terms of like two to three year horizon. I really think games are going to become these AI systems. It will be unlikely that sort of like when games transition from single player to multiplayer, I think that that new element of the AI system in a game is now going to be, is going to be table stakes. Being able to add your own characters, modify characters, actually affect the plot of the story and have everything else, the plot of the game adapt around that. And then being able to make like create your own games and then easily make them multiplayer and share them with your friends so that your users of a console or your users of your game itself become the creators, become the sharers and the best ideas rise to the top. And win. Chris, any final thoughts? And then I think we're going to take some questions here.
- [49:32]
Chris
Yeah, my only final thought is I think that one of the things we focus on in our podcast and our product is we really want to look at what is possible with the AI systems now. Like, we're always looking at what's practical and what can be done now. And I think everything we've shown today is possible now. In fact, there's more. More possible than we. We showed, obviously. And so I think that that's really the attitude that I think will work in the short to medium term, which is let's. Let's use it at what it's good at now. Yes, let's talk about the future possibilities and stuff like that. But I think focusing on what you can actually do with it is the thing that is, like, delightful and fun and interesting, and that's what we care about.
- [50:33]
Background Song
I am a background song you barely notice I'm here Composed to be ignored while you fall focus on the game but between us I think I'm pretty good this is a background song.
- [50:56]
Mike
Not.
- [50:56]
Background Song
Too short, not too long Am I setting the right mood? God, I hope I don't intrude Just a background song doing my best to belong Sometimes I wonder if you mute me Please don't For that podcast you've been meaning to hear I spent weeks on this chord progression but you're focused on collecting that gear it's fine I'm not crying, I swear do the other songs get more attention? The boss battle theme thinks he's so cool and that menu music is such a show off While I'm just ambient and forgettable oh, no, was that too needy? I'll just be here in the background Looping until the end of time not too catchy to distract you not too boring to be a crime Just existing musically this is a background song not too short, not too long Am I setting the right mood? God, I hope I don't intrude Just a background on doing my best to belong Please don't turn me down I have feelings too, you know.