Loading summary
Transcript38 lines
- [00:00]
A
Okay, you've probably heard of GLM 5.2 that's going viral everywhere on Twitter. Yes, it's this new open source local AI model that people are saying is the chat GPT moment for local AI, but no one's actually gone and shown you how to use it and how do you actually set it up? So I figured I'd bring on my friend Amir. He tells us exactly how you should think about running GLM 5.2, how you should think about running local models, how that integrates to something called Open Router, how you can use it with your codex or cursor or Claude code. And this episode, in 20 minutes or less, you're going to get everything you need to know about local AI models, why GLM 5.2 is crushing benchmarks, and how you can set it up today so you can go and build your startup, build your business and be more productive and be more efficient. Enjoy the episode and I'll see you at the end of it. Hit a like comment and subscribe for more of this sort of stuff in your feed. Enjoy. Welcome to the show, Amir. By the end of this episode, what are we going to learn?
- [01:15]
B
We're going to talk about, essentially learn about how local models are kind of keeping up now with the pace of these closed models as well and how you can kind of use compounding models or fusion models as Open Router calls it, to be able to do sequencing between a more extensive thinking model and a more execution based model. We'll show you how GLM 5.2 actually compares and stacks up against other models and how you can effectively use it and get set up with it as well.
- [01:41]
A
Cool. I did an episode on local models. It was a hit. People wanted more tactical. How do I actually, you know, how should I think about local models? How do I implement local models, How I can actually make this a part of my daily workflow. So I brought on Amir. Welcome to the show and let's. I give you two welcomes by the way. That's how excited I am to have you share with everyone, everything. So let's get into it.
- [02:07]
B
I'm super excited to be here. Let's jump right into it. So let's talk about what happened this week. Zai came out with JLOM 5.2 and I think this was a big inflection point because we've typically seen with local models, it's either, you know, storage extensive that you can't essentially run it or install on your computer, or you need a better like GPU RAM performance to be able to actually run it locally as well. Now GLM 5.2 is also resource extensive, but what we're seeing is with open source models like open source providers like Open Router or llama, being able to help you run these models in the cloud and effectively being able to essentially pay slightly less for input and output tokens compared to the more closed models. Now what I want people to take away from this session is one how to actually get set up with it. We're not going to go through the detailed setup process, but I'm going to just cover how you can do it in cursor, using Open Router or in Codex and then effectively talk about how GLM 5.2 stacks up against the other models and then how you can effectively use it as well to do model training. And then we'll do model chaining and then do just a kind of a, maybe a quick walkthrough of how I'm currently using it. You know, I want to be very honest, like these local models still have a lot of work to do in terms of having tool capabilities to be able to, you know, have the, you know, the modalities to be able to see images and conceptualize on what they're looking at. And I'm going to tell you how you can effectively circumvent some of that where you can use other models to explain what the image is back to GLM and then have GLM work on it and also just have a very live test on how this stacks up against other models. You know, benchmarks are great. Personally, I'm not an expert in it. I don't know what any of these benchmarks actually mean. The way I do it is off ops like let's build it out and see how this actually looks and how it stacks up against other models. Sound good?
- [04:00]
A
Yes, sir.
- [04:01]
B
Let's do it. Okay, so GLM 5.2 came out and essentially it has a 1 million context window and it scores 81 points on the terminal bench 2.1. It's just about 4 points behind Opus 4.8 and it does quite well on the long horizon task evaluation. So this is essentially projects you have that you want to run long sequence tasks on. And you know, I think it's taking account like the thinking parameters and how it can think through and plan through some of the, the task it has at hand. So you can see that across all these different kind of benchmark reporting reports, GLM 5.2 actually does quite well. So in this case it's 62.1% compared to open system 69.2. What's special about GLM is it's open source. You can run it locally on a machine if your device can support it, or you can run it in the cloud through the open model providers. It's a big leap from 5.1. I personally didn't test 5.1. I got straight into 5.2. But from what I'm seeing based off of like Twitter and conversations with people, it's performing quite well, especially on the front end side of just execution based tasks. I haven't really tested more on the backend resource, extensive tasks, but just based on perception and what we're seeing in the reporting, it's stacking quite well.
- [05:23]
A
So when you say stacking well, are we talking like. Because like I look at benchmarks and honestly it goes through, you know, one ear or one eye and it goes out the other eye. I'm like, I glaze over it. Cause I'm like, what does this really mean? Like, are we talking, is it like 4.8, is it like 5.5? Like what do we, how should I think about this? And yeah, give it, give it to me straight.
- [05:46]
B
Honestly, man, I'm gonna say, man, I don't get it. You know what I mean? Like I'm, I'm not smart enough to understand how these benchmarks actually stack. So I'll be honest. Like for me it's like let's just, let's just build it, use it and see how we feel about, you know, how it performs compared to the other models. And for me I want to get the best out of it. So if I feel like JM 5.2 is strong in one part but weak on the other, then I think about how do I actually use other tools or other models to essentially now I think like almost like a fusion approach. Now I think Open Router, which is one of the, like the model providers coined this where it's like you're able to do like sequencing between two different models to get the best output. So I'm totally game on if I can run a local model on my machine to do certain tasks, but then call, you know, Opus or Codex to do something else and have them work together. By all means. I want to be the most token and cost efficient and performance as well.
- [06:41]
A
Yeah.
- [06:43]
B
So on the setup side, I personally started using this through cursor, using Open Router's API. So how it works essentially is you gotta go to Zai, which is the GLM provider. They created the GLM 5.2 model, you get an API key from them, and then you take that key and you go into your cursor settings, paste it into the OpenAI field, and then from there you override the OpenAI endpoint with this API endpoint right here. And essentially from there you go back to models, add a custom model, GLM 5.2, and you're able to then actually call GLM 5.2 directly. So in essence, instead of OpenAI key, you put your GLM API key from Zai AI. Then you override the API endpoint for when you call OpenAI chat completion with this one right here. And then you go back into custom models and add the custom model protocol. You can also alternatively do this using Open Router. So if you want, if you're using Codex, you can go to Open Router and get your openaur key and then go into the provider, get the endpoint, and then go into Codex, create a profile and say, hey, I want you to install this model. Open source model. Codex does actually support open source models. So you're able to provide the details of what the model is, the context window. And then essentially, when you're running codex through the CLI, you can switch to GLM 5.2.
- [08:12]
A
Easy enough?
- [08:13]
B
Yeah, easy enough. You know, and maybe we can have a page or something to show later on where they can kind of follow these instructions. There's a lot of guides on Twitter and online, you can follow those. But essentially, in my opinion, the best way to get started with this is just go to Open Router and Cursor and get that set up through and through. So let's talk about here, the model. We've talked about, the benchmarks, how it performs, really, if we want to just at least takes, you know, put some weight to the benchmarks. I'd say if they're scoring at 62% and Opus 4.8 is at 69, you know, that probably means something. You know, for the normal people, we probably won't really know until we actually play around with it. But I went in and was just looking at, for example, this website. We have this small app and I built this in, I think, Opus 4.8. I was just testing it around and I started refining the design using J LOM5.2. So I was like, hey, redesign the Hero section for me or refine it. There's this like, section right here with all these images. I was like, why don't we just do a little, like, carousel style? So it's fascinating in a way, because I personally don't think open the local models had this kind of capability to be able to get it so refined and accurate previously. And I tested the models that we had, um, and I find that GLM 5.2 is a lot more refined and it's able to follow the instructions on, like, what you want to do. So in a couple of products, I was like, hey, you know, let's do a carousel here. Let's make sure we, you know, are able to show the images. And then from there I want you to build out like a bento grid style of all the features that we have now, this is all in one prompt. Obviously, you know, you can see it's a little bit vive coded here. He has a little side like badge the labels and you can tell. But at the end of the day, for a local model. For a local model, if you're running this on a computer and not burning any tokens, it's doing quite well. And I think I can see how it stacks in terms of reporting.
- [10:17]
A
I think. So. Okay, what is the main benefit of using a local model versus something in the cloud is you don't burn tokens. Essentially, the way to think about it is you're buying a machine and correct me if I'm wrong, but you're buying a machine and we should talk about some of the machines that people could potentially buy. But you're buying a machine, it's one size, it's a cost, it could be 2,000, 5,000, 10,000, and then you can just run. So you're building a startup, you just want, maybe you want conversion rate optimization. So maybe you say, every day I'm going to feed you customer feedback and every single day I want you to work on the front end and you just do that. So my question to you is for local models, specifically for people who actually want to be building companies, shouldn't they be basically running it all the time on certain tasks and how should people be thinking about it?
- [11:22]
B
Yeah, so I think that's a little bit tough to answer and I'll say why? Because, like, this model specifically is really resource intensive. So a lot of. I think existing consumer computers may not be able to run this. From what I've seen, I've been running it on the cloud through OpenRouter directly.
- [11:43]
A
And what's the cost with that?
- [11:45]
B
Yeah, so I actually was trying to map out the token cost of model training. So if we had about 50,000 input tokens and 85,000 output tokens to get almost close to an Opus 4.8 level of output it will cost us 44 cents. Whereas with Opus 4.8 it costs you $2.38. So there's a big, almost like a big difference on almost like 5x price difference between.
- [12:17]
A
Which doesn't sound like a lot when you're like oh, $2 here, 44 cents here. But when you're actually using these things and you're run all the time and you have pretty big tasks that you're going after and you don't want to be constrained by token costs. So it's a big deal. 5x is a big deal.
- [12:36]
B
Yeah, yeah. And this is based on kind of the averages on the coding benchmarks and what cursor charges you through the API pool. But I want to be also future proofing ourselves. If we got this far with J 5.2, I wonder what next 6 months looks like. Right. So would it be even worth. I think it's also worth thinking about making the upfront investment in your machine right now to be able to potentially download and run these local models so that when we get to GL 5.3 or 5.5, we're essentially made that upfront investment in the compute and the equipment to now save a lot more in the long run with other feature models that are going to come out that are going to be much more extensive. Because I think this, we're seeing the AI subsidy on tokens, right? Like we're getting a lot more output out of Claude, out of Codex. And I wonder, you know, I've personally seen it, I'm sure you have too, where it's like now we're hitting our usage a lot faster than we before. Especially when Fable came out. I remember like I ran it and like in the first day I hit my limit, you know.
- [13:36]
A
Totally. So what you're saying is basically like, you know, if you look at, I mean if you look at the history of VC backed startups, think about Uber. When Uber first came out, they actually subsidized rides and they got you hooked onto, onto the, onto the app and then over time they started increasing prices. Increasing prices. What you're saying is with,
- [14:00]
B
you know,
- [14:00]
A
in the AI age, with a lot of these LLMs, they're going to get you hooked into the workflows, you're going to build on top of it and over time, you know, those subsidies are going to go away as they go public and things like that. So what you're saying is maybe it's a good idea to actually invest in running this thing locally because the price of memory isn't getting cheaper and the price of tokens aren't getting cheaper. So building it now, securing it while you can, might be a good idea.
- [14:34]
B
Yeah. And to tie this all together, I'd say two things. One, harnesses that are agnostic on models. So like for example Cursor, where you're able to run multiple models across the same sequence of tasks, are going to actually potentially benefit from this. Right. So, you know, I wouldn't be surprised if one, you know, cursor decides to directly support GLM 5.2 as a model provider and lets you kind of tap into that cost saving if you couple it with like composer 2.5. So this where it goes into model training. Right. Where I was for example, looking at earlier in this task here, I wanted to refine the HERO section. So what I did is I actually used Opus 4.8 to, to first import screenshots because gel on 5.2 doesn't support vision capabilities. So what I did is I actually used Opus 4.8 to import screenshots and explain back to me what it sees. Right. I was like, tell me what you see specifically on the front end design for the HERO section and lay it out. And then I switched to GLM 5.2 to study that layout and, and then actually act on making those changes. So it's kind of a way to circumvent the fact that you have limitations to what GLM 5.2 can do on image capabilities. But you're able to kind of now train the expensive model to think through the plan and then get the same level of frontier level quality, but at a much affordable price point.
- [16:09]
A
I mean, that makes sense. So your, your recommendation is basically, you know, there's, it's almost like free trade versus protectionism. You know, the world you went, you, you know, not to be polit. This isn't political, this is just an economics theory, right? Which is like, you know, when, when, when people are trading with, amongst each other, you know, maybe it's, you know, in Canada where you are, you know, you might want to trade with Florida because, you know, we got good oranges here and we might want to trade, you know, we, we don't get, we can't make maple syrup here. So we'll get your maple syrup, right?
- [16:46]
B
We'll get a milk maple syrup. Yeah, exactly. So make the best use of it.
- [16:49]
A
Yeah, make the best use of it. You're saying is, you know, using cursor and, and basically, you know, you don't have to use cursor. Right. You can use whatever you'd like.
- [17:00]
B
Codex, clock code.
- [17:01]
A
Yeah, Use one of those to basically say, okay, for certain tasks I'm going to be using local models. For certain tasks I'm going to be using the best in class cloud models. And then together, ultimately you're getting great results in terms of the output, but you're also not spending through the wazoo. And if you're a token maxi like you and me are, in the sense of we're always pushing into the limit around anything we're building to get the most out of AI because you know, we don't want to hire 100 people, 500 people and stuff like that. It's helpful to do that and.
- [17:37]
B
Exactly. And anecdotally on two parts, right. One internally within our company. You know, I think Satya at Microsoft mentioned how like human capital plus token usage is now a big factor into what they're doing. Right. A lot of companies are now moving away from, you know, having direct access to the cloud code API to run the tokens because of how expensive it's become. Right. So they're canceling subscriptions. So we're seeing this firsthand with a lot of companies now are saying, okay, cool, this first year was great. You know, we had the mandate, you know, AI adoption, token maxing, you know, that's how we're going to measure success and that's how we're going to become AI native. Now they're like, wait a minute, okay, cool, we've done, we've done this. But we're spending way too much money on tokens. How can we now be more effective? Right? And I'm seeing this firsthand too where it's like, especially now, right? In a way you can have some sort of direct ROI between the tokens you're spending within the engineering team because you're like, okay, cool, we're saving a lot of time. There's an output, you know, engineer's expensive, we get that. But now you're providing the same level of harnesses and models to the non engineering teams that you know, are one shotting like, hey, help me format this email. And they're using Opus 4.8 high thinking. You're like, maybe that's probably not the right model. And that's a governance issue, right? That's a big thing. And I'm having these conversations with companies right now where they're saying, hey, can you help us figure out like how to build governance and proper education on how to actually use the right models? And this is where I Think model training is a big factor. Right. By the way, you know John at marketing, maybe you shouldn't use OPEX 4.8 to run this like to just format this email for you and helping them understand that. And I think, you know, I wouldn't be surprised if in a year from now companies start. You know, we've been thinking about it as well. We're like hey, why don't we just get our own machines and start running some local models? Because it's a lot more effective, especially how much money we're spending on tokens.
- [19:23]
A
What's the like? Just to play devil's advocate, why wouldn't I just use Open Router and call it a day?
- [19:30]
B
Please do. Yeah, yeah, absolutely.
- [19:33]
A
I think they should because you know when I'm on on X I see a lot of people being like buy a Mac studio or buy you know, these expensive devices, you know, for people listening, do they need to buy a local piece of hardware like if the price even goes up 2x or should they just use Open router and cursor or Claude co whatever harness they want.
- [19:58]
B
If I just have this right machine I'll get to this result. That's not how it works. No, you don't need a Mac Mini, you don't need this equipment. You can get started today. And what I love about Open Router and, and all these other tools is again they're so agnostic, they make it easy for you to be able to access this in the cloud. They run the models locally and it's credit based, load $20, get it going and easy to set up. I highly, highly recommend if you're starting to dabble with this and token usage is a thing for you, get one of these agent harnesses set up. Now that they're all, a lot of them are model agnostic. Run some tokens in open router, get these open models in there and start, start vibing. Just get some like I love to experiment to see how far I can take this. What if I plan with opus, review with GLM 5 point, execute with 5.2 and then review with Composer 2.5 or Codex 5.5. There's a lot of ways and I think we can be really effective and I think that's what the smart people are going to be doing in the near future.
- [20:55]
A
Some people are saying I don't care how much tokens cost because I think there's so much opportunity in building startups and optimizing and AI arbitrage that I don't even care cost Me, Whatever. What do you say to those people who are just basically ignoring this whole open source, local AI movement?
- [21:16]
B
I used to be the exact same person in our first episode. You're like, yeah, how much does all it cost? I'm like, I don't know, man. I'm just vibe spending. And I think that mentality has changed now. I can see that my usage limits are being hit faster and my cost is going up. And now that our team is expanding internally as well, so I think as a solo person, it's a lot easier to build a case or rebuttal around. Why you should just token max as much as possible, which itself is kind of like an. It's ironic. You shouldn't be token maxing. You should be token minimizing as much as possible and output maxing instead. So my answer to that is, if it works for you and you can directly have an ROI that you can show that, hey, I spent $200 and got a thousand out, great. Otherwise, sooner or later the subsidy is going to run out.
- [22:01]
A
All right, well, I think that's the episode. Unless there's anything else you want to add before we announce no.
- [22:06]
B
Yeah, I mean, for people that are trying out, dabble with it, play around, have some, see what it can do at least in the front end for you, and start working back in tests. And yeah, I hope they learned something from this.
- [22:17]
A
I'll include links for where to follow Amir. He's always one of my first calls whenever I'm trying out new stuff. And so I'm happy that you were able to jump on. I appreciate you. We it appreciate. Appreciate you. Give them a follow like and comment this video. Let us know what you think. We'll be in the comment section just out there trying to help and learn. And thanks a lot, Amir. I'll catch you on the next one.
- [22:44]
B
Thanks for having me.