← All TWiT.tv Shows (Audio)
All TWiT.tv Shows (Audio)
Hands-On Windows 189: Running Local AI on Windows 11
00:20:51
A First Look at Running AI Directly on Windows
Loading summary
Transcript7 lines
- [00:00]
A
Coming up next on Hands on Windows, we're going to take a look at a topic I've wanted to cover for quite some time. Local AI
- [00:09]
B
podcasts you love from people you trust. This is Twit.
- [00:19]
A
Hello, everybody, and welcome back to Hands on Windows. I'm Paul Thurat and this week we're going to take a look at local AI. This is running smaller AI models on your computer instead of bigger AI models up in the cloud, which is the more typical situation. So if you are familiar with something like cloud here, anthropic cloud, which I have running in a browser, in this case ChatGPT, Copilot, etc. Those things are typically running in the cloud and they're very big AI models. They're expensive to run, they're incredibly powerful. But there's another type of AI model, small language models that can run locally on your device. There are models that are small enough to run on a phone, models that can run on computers, et cetera. And I've looked at this again and again and again. Obviously there's a big gap between big LLMs in the cloud and small SLMs on your device, but the gap is shrinking as the quality goes up in both cases. And there are some big advantages to running AI locally. The most obvious. Well, I guess maybe it's not the most obvious one, but one of them is that it can work offline. It doesn't cost anything. Right. Depending on the capabilities of your computer, you might be running against the cpu, which can be resource intensive. You might be running against the gpu, which could be fantastic if you have a good one. You could be running against the NPU if you have a Copilot plus PC or an AI PC. And that can also work very well. And, you know, there's some complexity to it, but it's. It's actually very, very interesting. So there are different applications you can use to do this. Some of them are very technical. For this I'm going to use something called LM Studio when you. But this is one of probably hundreds of choices. This is cross platform. You can get this on the Mac. I think it's on Linux as well. And when you first run it, it asks you if you want to download the model it recommends, which is actually what I did download, but it's the one I wanted to download anyway. Which is this Gemma 4e4b. And yes, that's kind of a strange name, but Gemma is the SLM or local AI version of Gemini, the big Google models up on the cloud. It comes in different versions. This one, if you look at the name of the thing, it's the fourth generation of this model family E. 4B. Refer the 4B. Well, the E is efficient and there are not. There are non efficient versions as well. Meaning the alternative is active and it has to do with how many parameters, which I'll get into in a moment. Sorry, there's a lot of terminology here. Can be running at the same time and then the 4B is 4 billion and that's for parameters. It's sort of a waiting system for the number of internal variables that the model can consider as it's doing its work. It's kind of like synapses in your brain or whatever. So I've already loaded this, I believe. Yes. Because this eject button is how I would get rid of it. I can look over here at the models on. I have installed this computer. I just have the one I should say. Gemma is a multi model, sorry, multimodal model. Meaning it can work with different inputs, including images and text. It only outputs text. There are local models that can output images and other types of content. I've been experimenting with those as well. But I think for this first look at this, I'm going to stick to text. Let's keep it simple and we'll look at images and other things later. You can also go in and just see what they have or models. The one thing I'm not 100% sure of on this particular application is whether you can search for models that specifically run across the GPU or the mpu. But I know, I know in this case, I assume it says it somewhere here. I don't see it. But this can run at least partially against your gpu. Some of these models will say, literally it can run 100 against the GPU and that's actually a very efficient, in terms of time way to do that. Meaning it will be faster and it can run off the gpu, but it's not. I'm actually not clear where this is running against, but it's going to be cpu, GPU or MPU or some combination of those things. So when we bring this thing up, it looks like, you know, again, we can go back to cloud here. It's very similar, right. You've got this kind of bar over on the side with different options. You've got your previous conversations, you can start a new conversation like this. It's set up to do thinking by default. This is reasoning. This is kind of an interesting thing because this is the type of AI model where you can see its internal reasoning, if you will, as it tries to build the answer to whatever it is you've asked it. Some people don't like that. To me, it feels a little bit like stalling for time because it does take some time. You know, these models are slower and they're, they're not as powerful obviously as the things in the cloud, but there's still some really neat things you can do there. So let me go through some simple prompts and some things I've tried in the past a little bit so I can kind of understand, you know, I have some idea of what's going to happen and I'll show you how this thing works. But by and large, you know, this should be very familiar. This is a lot like Copilot, chatgpt, cloud, whatever. Yeah. Let's take a quick break and we'll be right back.
- [05:45]
B
Hey, let me interrupt just briefly to tell you this episode of Hands on Windows is brought to you by Threat Locker. Threat Locker Zero Trust platform now delivers the industry's most comprehensive suite of zero trust solutions, protecting endpoints and networks and the cloud. Yeah, they've extended zero Trust enforcement to cloud services and company networks. That means now Threat Locker ensures that devices are validated through a secure broker before they can connect to cloud platforms like Salesforce, Microsoft 365 Asana, Google Workspace and GitHub. And you know what that means? It means even if your employee, if a user is successfully phished, even if they bite and they get hacked, attackers still cannot access those resources. They actually have to have physical possession of the user's trusted device and authenticate on that device. That ain't going to happen. ThreatLocker works across all industries, provides 24. 7 US based support, the best support in the world. They support Windows and Mac and Linux environments. And with ThreatLocker as just a bonus, besides the great security, you get comprehensive visibility and control. It's great for compliance. Just ask Rob Thackeray. He's end user technical architect at Heathrow Airport. Man, I mean, talk about an important job. He said this quote, threat Locker was the most intuitive solution we tested. And the responsiveness of the organization, the willingness to engage with us, set up a demo and work with us on weekly audit reviews was very good. It's great to have an ongoing relationship with the company that's so responsive to our requests. Threat Lockers trusted by Global enterprises like JetBlue, the Indianapolis Colts, the Port of Vancouver, Threat Locker consistently receives the highest possible honors by all of the, you know, major players. In industry recognition like well, G2 gave it their high performer and best support for enterprise. Summer 2025, Peerspot ranked threat locker number one in application control. And GetApp gave ThreatLocker their best functionality and features award in 2025. And I can go on and on. Threat Locker lets you confidently ensure users have access to a consistent, safe network connection. That means offices, remote users, internal servers and critical services can maintain smooth operations without the need to open inbound ports or deploy traditional VPN solutions. But your end users will get the secure, reliable internal system access they need no matter where they are without complex infrastructure changes. Get unprecedented protection quickly, easily and cost effectively. Really cost effectively with threat locker. Visit threatlocker.com twit. You'll get a free 30 day trial. Learn more about how ThreatLocker can help mitigate unknown threats and ensure compliance. That's threatlocker.com twit now back to Hands on Windows.
- [08:43]
A
So the first one, it's, you know, I'll just type this. Can you tell me how many people used each major version of Windows? I'll, I'll eliminate the suspense and tell you. It cannot tell me this, but we'll see what it does of Windows over time. And this is where the thinking starts. So you can see it. This is it sort of talking to itself, right? You can click on this to expand it so you can see more of it. And as it says here, it's difficult because we really don't have good data for a lot of this. It's not clear how many hundreds of thousands of people use say, Windows 2.0 or whatever. Microsoft at some points in time has, you know, provided data for the number of people using Windows, et cetera, but sometimes not so much. So this is thinking through the problem, so to speak. I can't see. Yeah, if you look at like something like it says, focus on the revolutionary nature, not massive numbers. It's not really talking to me there, it's talking to itself. That makes sense. But now you can see this part out here. Is it supplying the answer? The other thing, as it does, is to look at is this little number and circle down here. If you hover over this, it tells you that this is the number of tokens that this thing is expending. And there are tokens that kind of go in and there's input tokens and an output tokens. The way to think of this is that a token is a unit of work essentially. It's kind of the currency of AI input and output literally in the cloud. It's literally the currency because that's how you get charged now for using AI. And then there's this. Let me hover over that again. This notion of. Well, it actually doesn't say context here. Oh no, it does, yeah. Total load of context 4096. So this particular model has a context window. Another great terminology of 256,000 tokens. And that's what the AI can essentially remember in a given conversation. If this thing goes over 100%. And it won't in this version or in this query, but it will actually just forget the conversation that goes past that number. Right. And so if you're within 100%, this is. Most of the things I'm going to do today are about 50% of the context window. What that means is if you keep asking it questions, it will remember based on what it already did, but once it exceeds a hundred percent, it will start forgetting kind of a Flowers for Algernon kind of a thing there. Okay. So it has provided this information. It's nicely formatted. It doesn't literally, I don't think anywhere say X number of million or billion people used in particular version of Windows. But I do know for the recent versions like 10 11, Microsoft has said explicitly over a billion in each case. And they don't really have that because I already knew they weren't going to do that. But. But still, you know, very interesting. And you could dive in, you know, further here if you wanted to. I'm not going to bother with that. This box up here is the developer interface, and that's an optional feature you can enable when you install the app. I did, and I did enable that. You don't need that to ask IT developer questions, but instead of an actual code question, in other words, here's some software code, tell me what's wrong with it, or how could I make this more efficient, etc. I'll just ask it a coding question. So, you know, can you tell me what a singleton is in programming and whether it's. Well, I will go with that. I'm going to say whether it's good or bad to use it. Um, generally speaking, it's bad to use it, but we'll watch it kind of think through this. So singleton, if you're not a programmer or are a programmer, sort of like a global variable, static variable language, it kind of breaks the whole object oriented programming paradigm, but they're super convenient to you. So people use them all the time. Developers do, I should say. But again, this is it. Thinking through the problem. It's going back and forth with itself, essentially with the data it has access to. And now it is providing the answer. So yes, it is often debated in object oriented programming. It's explaining what it is, etc. So you get the idea. This past weekend we were away and I was experimenting with different models and different apps and. And I was trying to think. I want to do something complex but not technical, you know. And if you watch Twinders Weekly, you'll know that JRR Tolkien has come up a couple times recently for whatever reason. And I jokingly often refer to myself as a Tolkien scholar. And so I thought, I'll just stop that. In fact, what I should be doing here is a new conversation for each of these things was I'll ask it about Tolkien. You know, like, I know a lot about Tolkien. I've read those books dozens and dozens of times. There was a period of time where I read the Lord of the Rings every single year, you know, for example. So I'm going to pretend I don't know anything about it. I want to read Tolkien. Where should I start? And are some of the books better than others? The lack of a better way to frame this. So once again, I'll just expand so you can see it's kind of thinking through this. If you're familiar with Tolkien at all, you'll know that the Hobbit is the obvious place to start. It's super, you know, it's a. It's almost a children's book. It's very short and easy to digest. The Lord of the Rings, of course, is the most famous one trilogy of books that everyone has heard of and probably seen the movies. And then there's the Silmarillion and all the background works that have occurred since his death or have been published since his death. I should say Silmarillion is like reading the Bible, meaning it's not a good read. It's more of a kind of a mythology history work or whatever. So let's see what he says here. Or I've already turned it into a human being for some reason. But yes, so, so where you should start, obviously the Hobbit, you know, lighter, quicker, etc, Lord of the Rings, I assume the Silmarillion will be third or whatever. Doesn't even get into the. Oh, he's going through all those books. Yep. That's interesting. Doesn't even mention that. That's fine, we'll just move along from that. But when I was doing this over the weekend, I, I kept going with it. You know, well, what about this? What are the major themes? What are the, you know, what, what's the point of these stories, et cetera. And it's fascinating to watch the local AI, especially because something like this, it can quickly blow through its context window and it just forgets everything it ever said, you know, to you. But I kind of want to see where it's at. I'm going to guess this is going to be a little over 50% just for this little report style answer it gave. Yeah. So this thing will eventually catch up. It's not because it's clearly not 1%, but maybe it is 1%. Oh, there we go, 44%. Okay, you can do other things with this. Obviously you could upload a document. This has not worked for me on this computer, so I don't want to embarrass it or me, but you can see the various qualifications here. But when I was doing this on a more powerful computer, I originally tried to upload a PDF. I ran into size issues. I uploaded the smallest PDF I had for one of my books. It still was not happy with that. But I could go into my book folder here where I have the source files for those books. So, you know, an individual chapter, which is a markdown file for the book, should work. It's not going to, it's going to fail for some kind of a node JS kind of a problem. But I'll just ask it. Summary. Yeah, so normally what would happen is it would provide that kind of report like you saw before. I don't understand why this is failing on this particular computer. So I don't have node JS or whatever, but I feel like that's something that maybe another app would have handled a little better. Similarly, you could use this to analyze an image. Again, I can't output an image. If I. In fact I could just say, can you create an image? And I'll just give you the same error. That's interesting. Normally what it would do is just come up with an error message and say, this is the type of model I am. I can't do that type of thing. And you could go into here and then maybe search for a model that can do images. And interestingly this one does come up, but that's because it can process images in that direction. So you want to find one that can work for that and output images. But again, that's something we will look at later. So. So this to me is incredibly useful. You might still want to turn to cloud based AI for many things, but I Do feel like this is a good place to start this or another app and. Or another model, right? Microsoft OpenAI rather. Sorry I was gonna say chat, GPT Etc. These companies all have local models, deep seat, many, many of them. And then there's the cloud hosted stuff which you know is free and paid, right? Even if you pay for something like cloud or chat GPT or Copilot or whatever, it almost always makes sense just to start here. Especially if you're looking for help writing or want the answer to some complicated question etc, and if you kind of go past the capabilities of this thing, you could then move on to the paid or bigger cloud based AI. Plus if you're on a plane or you're just offline, it's kind of nice to be able to go back and forth and have these conversations because, you know, within the context of any of these things, literally and also in the language of AI, you know, you can see in this case it had used 63%. You could keep going to a point until you get to where you want to be and then if you exceed the context, you might have to give it some information that it provided previously, which is kind of curious, but to keep going. But you can keep going. And this is free, like literally free. So depending on what you're doing, I think this is pretty much just as good as most of the more famous, you know, cloud based. So it's something, something to look at. And we will look at this again in the future. I want to do an episode at some point where we use local AI to create images and other content, you know, things like videos and so forth. So we will get there, but this is the first step and then, you know, we'll take it from there. So hopefully you found this useful. We will have a new episode of Hands on Windows every Thursday. You can find out more about the show at TWiT TV. Ho thank you so much for watching. Thank you especially to our Club Twit members. We love you. If you're not a Club Twit member, you can learn more about that program at TWIT TV Club Twit and it really helps out all the channel and the company and all the content creators there. If you become a member plus you can see the videos without ads, you can join in as for some of the shows where they record them live and you can interact with other listeners and so forth. So thank you and I will see you next week.
- [20:08]
B
Hey everybody, it's Leo Laporte. Are you trying to keep up with the world of Microsoft. It's moving fast, but we have two of the best experts in the world, Paul Thurat and Richard Campbell. They join me every Wednesday to talk about the latest from Microsoft on Windows Weekly. It's not a lot more than just Windows. I hope you'll listen to the show every Wednesday. Easy enough. Just subscribe in your favorite podcast client to Windows Weekly or visit our website, Twit. Microsoft's moving fast, but there's a way to stay ahead. That's Windows Weekly every Wednesday.
- [20:52]
A
Some follow the noise. Bloomberg follows the money. Whether it's the funds fueling AI or crypto's trillion dollar swings, there's a money side to every story. Get the money side of the story. Subscribe now@bloomberg.com.