← All TWiT.tv Shows (Audio)
All TWiT.tv Shows (Audio)
Hands-On Windows 158: Semantic Search
00:14:04
How AI Is Quietly Transforming Windows Search
Loading summary
Transcript18 lines
- [00:01]
Paul Thurrott
Coming up next on Hands and Windows, we're going to take a look at a feature called semantic search, which may or may not finally solve all of our search problems in Windows 11. Let's map out this week's amazing destinations and travel tips.
- [00:15]
Will
Honestly, Will, I didn't plan any trips, but I did switch to T Mobile with their new Family Freedom offer.
- [00:22]
Paul Thurrott
That's not the itinerary we're following.
- [00:25]
Will
Well, I'm departing from AT and T and embarking on a new journey with T Mobile. They paid off my families four phones up to $3200 and gave us four new phones on the house.
- [00:38]
Paul Thurrott
Bon voyage.
- [00:40]
T-Mobile Advertiser
Introducing Family Freedom. Our lowest cost to switch our biggest family Savings all on America's largest 5G network, visit your local T Mobile location or learn more@t mobile.com familyfreedom up to $800 per line via virtual prepaid card typically takes 15 days. Free phones via 24 monthly bill credits with finance agreement eg Apple iPhone 16128 gigabyte 82999 eligible trade in eg iPhone 11 Pro for well qualified credits and balance due if you pay off early or cancel contact T Mobile Ever notice.
- [01:07]
Will
How ads always pop up at the worst moments when the killer's identity is about to be revealed during that perfect meditation flow on Amazon Music, we believe in keeping you in the moment. That's why we've got millions of ad free podcast episodes so you can stay completely immersed in every story, every reveal, every breath. Download the Amazon Music app and start listening to your favorite podcast podcasts Ad free included with prime the kickoff to.
- [01:41]
Joshua Tree Chrysler Dodge Jeep Ram Advertiser
Fall is on at Joshua Tree Chrysler Dodge Jeep Ram. Shop hundreds of vehicles all at the best prices in the desert and during the Drive into fall event. Take 15 grand off MSRP on a new 2025 Ram 1500 Crew Cab big hard lease a 2025 Gladiator Sport for only 249amonth for 24 months or lease a new 2025 Grand Cherokee Altitude for just $299 a month for 24 months. No hidden fees, just competitive pricing you can trust. Looking to trade? We want to buy your car whether you buy from us or not. Take advantage of the highest trade in values of 2025. Today visit Morongo Basin's exclusive hometown dealer Joshua Tree Chrysler Dodge Jeep Ram where the price is worth the drive or shop Joshua Tree cdjr.com with approved credit must finance Atlantis Financial Services 1995 or 2995 do it sign up see deal.
- [02:39]
Paul Thurrott
Over details.
- [02:42]
Joshua Tree Chrysler Dodge Jeep Ram Advertiser
Podcasts you love from people you trust.
- [02:47]
Paul Thurrott
This is Twit. Hello everybody and welcome back to Hands on Windows. I'm Paul Thorat and if you have been around Windows for a while like I have, then you probably remember things like Longhorn and WinFS and database spec file systems and how we were going to solve all those file search problems and then didn't. Yeah, this never happened. And so here we are, decades later, we're still using the same index based search functionality that we've always had. Windows added search directly to the start menu early 2000s. I don't even remember when that was. They added it to the taskbar, which is really the same Start in Windows 10 and Windows 11. And it's not very good. Right. If anything they've made it worse. Right. There are these weird interfaces, like, let me bring up search here, like the search highlights screen which has ads on it. And you know, it's just been kind of a whole lousy thing. But in recent years, AI has happened. And not just AI, but local AI. Right. So we have these cloud based LLMs like OpenAI, ChatGPT and Copilot. And Microsoft, like Google and Apple has also been working on local, smaller, smaller SMLs, I guess. SLM, small language models, AI models that run on the device. Right. And so in the PC space, this type of thing occurs today exclusively on what's called a Copilot plus PC that has an MPU of a certain type, I think this is going to expand eventually to other types of computers. But for now that's what we have. And it's kind of an interesting test bed for some of these new AI features that are coming down the pike, one of which is called semantic search. And if you do have a Copilot plus PC, you've probably seen this. So if I actually turned off my taskbar based search, let me put that back on so you can see it, I'll turn on the whole search box. So if you click in here and start typing, this will work like it has in Windows 11 kind of all along. But you can see here it's got this purple, blue, pink kind of icon which helps to indicate that maybe we're doing something a little different here or could be doing something a little different. So I could just do something simple like search for the name of an app. And because that app is on my system, I can click on it and it runs. And that works the way it did before. Right. The other way is just to go directly to the start menu. So you can either bring up the Start menu like you would. You can see up at the top that the search box is selected. And what that means is if I start typing from here, it will give me that. It goes right to that same search experience. Right. I could also do a keyboard shortcut so Windows key +S will bring up that search experience. Because I've got the search box down there I'm typing in into the taskbar. But if I didn't, there would be a search box up here in the top of the window. So so far, that works exactly the way it's always worked. But there are other places to search in Windows, Right. Obviously, you can go to the file system using File Explorer and same thing. See blue, pink, purple. Right. So we're playing around with what happens here, but the idea here is that you type for something. Typically what you would really do is go maybe to a particular folder and then maybe go into whatever and then search from there. Because this will kind of not ground the search, but filter the search down to what it is you're looking at, rather than the entire database of items it could search. Also, just on a side note, although this is not necessarily an exact topic for here, we also have Search in the Settings app. Right. And so one of the other features that's coming in windows 1125 H2 and 24 H2 is this notion of semantic search or natural language search really in the Settings app. And so they have a nice sample there. So my mouse pointer is too small is actually kind of cool because this is a plain or a normal English way of saying this thing. And instead of just matching it to an exact feature, it's using its natural language capabilities to understand what it is you're looking for. And it's even providing a way to kind of do it right there in line. Right. Which is nice. So that's. That's kind of a nice improvement. Let's map out this week's amazing destinations and travel tips.
- [07:20]
Will
Honestly, Will, I didn't plan any trips, but I did switch to T Mobile with their new Family Freedom offer.
- [07:27]
Paul Thurrott
That's not the itinerary we're following.
- [07:30]
Will
Well, I'm departing from AT&T and embarking on a new journey with T Mobile. They paid off my family's four phones up to $3200 and gave us four new phones on the house.
- [07:43]
Paul Thurrott
Bon voyage.
- [07:45]
T-Mobile Advertiser
Introducing Family Freedom. Our lowest cost will switch our biggest family savings all on America's largest 5G network. Visit your local T Mobile location or learn more@t mobile.com familyfreedom up to $800 per line via virtual prepaid card typically takes 15 days. Free phones via 24 monthly bill credits with finance agreement eg Apple iPhone 16128 gigabyte 82999 eligible trade in eg iPhone 11 Pro for well qualified credits end and balance due if you pay off early or cancel contact T Mobile and.
- [08:11]
AT&T Business Advertiser
Now a next level moment from AT&T business. Say you've sent out a gigantic shipment of pillows and they need to be there in for International Sleep day. You've got AT and T5G so you're fully confident, but the vendor isn't responding and International Sleep Day is tomorrow. Luckily AT&T5G lets you deal with any issues with ease so the pillows will get delivered and everyone can sleep soundly, especially you. AT&T5G requires a compatible plan and device coverage not available everywhere. Learn more@att.com 5G Network.
- [08:41]
Paul Thurrott
But the point of semantic search is to hope to hopefully solve this problem that we have with index based search, which is that at least when I in my experience when I type a search for anything, I never find it. So if it's in OneDrive, for example, I could search in the file system and I never find it. But if I go to onedrive.com and search for it there, I find it pretty immediately. And so what this does is it works off of the index that your computer is already creating and then it creates. It uses an on device AI model to basically create a what is in essence like a secondary index that's based on natural language search. Right. And so you can just type, it's searching basically a more constrained, grounded version of the index, if that makes any sense at all. So let's see what that looks like. And this will be terrible because I just turned this on in this computer this morning and it's not completely indexed. But let's give it a shot. So I mentioned Longhorn, so maybe that's a good place to start. So, because I have written a lot of content about Longhorn, or I did back in the day, so 20 years ago. Ish. And you can see here it's not quite done, but of course when I search For Longhorn, Windows 11 search by default is searching everywhere. So you're getting the best match is the Longhorn Steakhouse, which is absolutely not what I was looking for. But you can see over on the side they have. I guess these are. Yeah, these are reasonably close locations for this restaurant that I will never visit. It did find one folder that's kind of interesting. But I could go into Documents and just see this. And sure enough, enough I can tell just from looking at these things that these are articles mostly that I would have written, you know, a long time ago. Right. And so I'll give this one a shot. We'll just click on this, see what happens here if it ever loads. Loaded in a different window here. But yeah, here we go. So this is an article I wrote in 2005 that appeared in PC Worlds. Yes, PC World magazine, about 64 bit versions of Windows, which at that time were incredibly rare. But the point is, this is somewhere sitting in my OneDrive, as you can see from the URL and in my archives. And even though this file is not live, you know, it's not synced to my desktop here, it still was able to find it. Right. And so this is kind of one of those neat benefits of the AI in a way. It's not that index based search maybe wouldn't have found it, but you probably wouldn't have found it. So in my case, I find that I either have to navigate in and then search from a small, you know, like from a local location or from a location that's more confined or like I said, go to the web. So that's kind of interesting. It works from everywhere. So obviously if I do it from here, I get the same exact result. Right. And you can see the same thing. But you would typically do this type of thing or I would anyway, from File Explorer. Right. And so to me this is a little more interesting. So again, I'll just keep. I'll stick with the Longhorn example and this one I actually really like because I use Synology drive and I can see this result here, some Synology. I do have my archive up in OneDrive, which you can see from the cloud icon here is not, it's not local, it's. I mean you can access it locally, but it's not, you know, downloaded, but it still found it, which is kind of cool. Let's see if I could go into any of these that makes any sense. Let me see an article or a folder I should say from 2005 that just has, okay, it's just a web article, but let me go back actually. So. But you get the idea. So this came up, I mean, I wrote a lot about Long. So this thing found 374 results are still going actually. So there's going to be over 400 eventually and probably over 500 really. So in my, in this case, it's everything from screenshots to PDFs to articles that I wrote, Word documents, HTML documents, whatever, it's like all kinds of stuff. So it's like this whole history of whatever that's tied to that particular search query. So that's pretty cool. So the problem with this to me is only that you have to have a Copilot plus PC to get this. But like I said, I think it's going to come to everybody. If you're familiar with how the index works today, probably not most people aren't, but there is this old school indexing options control panel which you can find with Start Search and you can see it's actually, it's just going to town here trying to index the drive, which I'm sure it's not impacting the performance in any way, shape or form, but what you would typically do is click this modify button and then you could just see what all the locations were. I'm getting one of those user or Windows slow authentication things and then you could just kind of dive in. Right. And so in my case, like, I know if I go into my Paul account, oops, I don't want to turn that off. But you can see all the stuff that's in there and you could just kind of go through and you would just select like I want you to index this part of it. In Windows 11 there's actually a modern version of that UI in the settings app. And so if you go into privacy and security and then search, this is some of this. It's actually all of the same options with other options. Right. And so in this case you could turn off search highlights. And that was that UI that was so terrible. So you can see this is off, which is wonderful, which is what you actually want. And then for best results I found even though it has to do this initial index, it's set on Classic. By default I turn this on to Enhanced and it's basically the entire PC. Right? And what that means for me is that if I dive into my user account here in File Explorer, I have Google Drive, I have OneDrive and I have Synology Drive and each of those things, Synology Drive is the thing I work off of. So that's my kind of current work. But all three of these locations have some archives. And because the indexing and the semantic search that are available now on Windows 11 with a copilot plus PC can work with cloud accounts, which is an option somewhere. Where is it somewhere in here anyway? There's an option here somewhere that will say maybe it's not tied to that. I hope that will say, you know, search through your cloud accounts as well. So right now, OneDrive is the only thing that's technically directly supported, but it's an open API that anyone could use. And in my experience, it already actually works. So you saw the results from Synology Drive, even though that thing is not part of that kind of new cloud system that they have. So does this kind of solve the problem? Yeah, asterisk. I mean, the problem with it, of course, is that it only solves the problem for those of you who have a copilot plus BC, which is probably less than 10% of you. Right. So indexing still occurs, you still have to do that, you have to deal with that, so to speak. I mean, most people wouldn't touch it and it would just kind of work, but I found that it works better, like I said, when you turn on that enhanced indexing mode. And I think it's going to come, like I said, eventually to everybody, it has to, right? I mean, this is such an obvious feature. It's something that we have been struggling with in Windows for so many years. And this is, I think this is going to kind of solve it. It's interesting there was no one over the past 20 years who said, you know what, once AI comes down the pike, we're going to solve that problem. They've been working on different ways to solve it, but when AI did arrive, they realized, wait, actually this is going to work really well. And you know, it's tied into some of the features that are built into Windows 11, like click to do or recall where it's using AI behind the scenes to do pattern matching and understand what is in a document or is in an image or whatever, and creating its own index so that instead of just searching against say the file name or if it's known file type, like the file contents, it actually can search against AI's understanding of what is in those things. So it kind of broadens the amount of information that can come back. And in my experience, it actually works faster and better. So don't often say that about AI, but that's the way it works today. So. So there you go. So nothing is changing from a UI perspective, except for the cute little sparkly purple and pink stuff. Keyboard shortcuts are the same Windows key +s for Start Search, Control F if you're in File Explorer. But if you do have a Copilot plus PC, definitely check this stuff out. Like I said, it just, to me, it makes finding things that are in your file system or up in the cloud, just a lot faster. So I hope you found this to be interesting and useful. We will have a new episode of Hands on Windows every Thursday. Find out more at TWiT TV. How thank you so much for watching. Thank you as always to our Club TWIT members. We love you and you can learn more about Club Twit if you're not a member at Twit TV Club. Thanks. I'll see you next week.