Loading summary
Transcript23 lines
- [00:01]
A
What if I told you you could get opus level reasoning at a fraction of the cost? That's what we're going to see and test today when I take a look at GLM 5.2. This is our first of many reviews of Openway and Open Source models to see if we should all be paying the tax to Anthropic and OpenAI or if we can run these models locally and get the same results. Let's dive in.
- [00:26]
B
This episode is brought to you by Mercury Banking. Redesigned from the ground up, now with Command so you can just say what you need and the work gets done. I've always liked products that reduce the distance between knowing what you want to
- [00:39]
A
do and actually doing it.
- [00:42]
B
That's one reason I've been a Mercury customer for years. Whether it's sending money, managing cards, or checking in on the business, Mercury has always felt fast, simple and built for founders. What I love about Mercury Command is that it takes that idea even further.
- [00:57]
A
Instead of navigating menus or figuring out
- [01:00]
B
where a feature lives, you just say
- [01:01]
A
what you need, ask a question about
- [01:03]
B
your finances, create an invoice, categorize transactions, or send a payment. All from a conversation. Because Command is built directly into Mercury, every answer comes from your actual account data and every action follows the permissions and controls you've already set up.
- [01:20]
A
It feels like where software is headed
- [01:22]
B
less navigating and more simply telling a product what you want to accomplish. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC insured bank. Banking services provided through Choice Financial group and column NA members FDIC okay, before
- [01:40]
A
I get GLM 5.2 running against some benchmarks on my own computer, my own projects, I want to talk you through what exactly this model is. So GLM stands for General Language Model and this is a model that that's put out by the Beijing based startup Z AI. So yes, this is a Chinese trained model. Now this model is open weight. You may have heard that term before but not exactly known what it means. And all it means is that the trained model weights are publicly available for download, which allows you to run it on your own hardware. You can fine tune that model on your own data and you can kind of inspect how it works. Now the licenses for open weight models kind of depends, so it doesn't necessarily mean you can use it quote quote unquote for free. But why this matters for GLM 5.2 is you can self host this. So let's say you have one of Those chunky Mac studios at home where you want to run some models locally, you could Potentially run GLM 5.2 locally. You can fine tune it, which means you can adapt it to your purpose. And then the thing that people really love about these open weight models is they're cheap, you can run your own inference and so it can be much more affordable than API cost from anthropic or OpenAI. And, and two, you're not logged into a vendor. So if a specific vendor changes their API terms, you can actually change what provider you use and there will be a lot of inference providers. I'm going to show one that you can use to run GLM 5.2. Now why should you be paying attention to GLM 5.2? Well, this is what I'm hearing from the breathless AI bros and I guess again I am a breathless AI brother is that GLM 5.2 is getting people sort of opus level intelligence or GPT 5 point whatever intelligence for a fraction of the cost and the ability to be self hosted. If this is true, this is a very big deal. As we've seen, we can't always rely on the big model providers for consistent model access. And these hyper intelligent models from the frontier labs are getting very, very expensive. And so any models where open weights models are catching up to the intelligence of OpenAI models, anthropic models, especially for coding use cases, which can be quite expensive, is something to pay attention to. Now let's look at the benchmarks and capabilities of the model and then I'm going to dive right into actually using them. So a couple of things you should know about GLM 5.2 just to pay attention to is its context window is big, it has a million token context window, so that's sufficient. But one limitation is it only takes text in and only takes text out. So you can't put an image, you can't get out images. It is a text to text model. That is one constraint of this model. That being said, it has all the capabilities that you should expect from a modern day model interface. It has reasoning or thinking mode, it can stream its outputs, it can call functions, it can do context caching to make things more efficient and it can output structured output and use mcps. So at the end of the day this is a very capable model with the right ergonomics that we've gotten used to. Now what do we see from the market benchmarks on this model? So as you see here on Frontier Sweep, Post Train Bench and SWE Marathon, it's inching up there to Opus and right above GPT 5.5 on a lot of these benchmarks. And if you look at SU Bench Pro, you can see it's about on par with GPT5.5 and almost up to Claude Opus4.8 certainly beating Gemini 3.1 Pro. So if you look at it against these models that we've all come to know and love, it's. It's in their arena. It's definitely worth testing out. And the external benchmarks say this is a model with enough intelligence to attack some of our hardest coding problems. So what I'm telling you is you can get this open weight model where you can inspect how it's actually built, you can run it locally or at least more cost efficiently on your own inference or your selected provider's inference, and it's going to code just as well as Opus 4. 8. Let's do it. So how do you actually get GLM5 in your coding stack? Let's say you're completely new to all this open weight model stuff and you want to figure out how to run these in Claude code or in cursor. I'm going to show you Claude code and cursor. It basically applies to codecs as well. I'm just going to give you those two examples because I think that's going to cover most of your use cases. So first you need to choose where you're going to get your model from from. And I'm still using a hosted API version of GLM 5.2. My little laptop's not going to run this thing locally. And so I've chosen to use Open Router, which is a unified interface for getting access to a lot of different models, both commercial and open weight and open source. So I signed up for an open router account, and then all you need to do after you sign up for an open Router account, give them your credit card, set a limit if you want to, and just set up an API key. So I set up a local dev, actually gave me another API key, so I set up an API key and now I have access to this model via Open Router. And so I'm going to show you two ways to set this up in cursor and then in Claude code so you can start using glm5.2. Okay. Setting glm 5.2 up in cursor is super easy, though it took me truly about 30 minutes to figure out the nuance here. No one has documented it. We'll put it in the blog post in the show notes for you but all you have to do is go into your cursor settings and click the Models tab. And then you need to do two things. First, you need to put your API key from Open Router Here in the OpenAI API key field and toggle that on. And then secondarily, you need to override OpenAI base URL with this very specific URL. So it's Openrouter AI API v1 cursor. I could not find anything for a really long time that told me it had to be slash cursor, but it is slash cursor and you need to toggle that change on. The second thing you need to do is add Z AI GLM 5.2 to your models. So you simply click View all Models, you add a custom model, you add that field in, and you will be able to access this model. So if you open up your chat in Cursor down here in the bottom, Z Dash, Aiglm5.2 is now available as a model running through Openrouter. Now that's cursor, and we're gonna come back to this. On claude code, there is luckily a little bit more instruction on how to do this. And so there is this page on Open Router, the docs page that shows you how to connect claude to Open Router. And then I'll show you how to connect your specific model. For cloud code, it's pretty simple as well. You need to get your Open Router API key and that URL which they have here, and you need to add to your shell profile. So for people who are not super technical who have just claude coded their way into Terminal, your shell profile is the file that manages your settings in the terminal. It's going to like instantiate a bunch of environment variables, it's going to set a bunch of settings. And so you need to edit your shell profile. It's usually zshrc or bashrc, depending on what profile you use, and add these lines right here to that file. You can also this says open it in Nano, if you're not feeling fancy, you can just find this file in your finder in your file directory and open it in whatever code editor of choice and add these lines which include your Open Router API key, the base URL, which does not contain/cursor. It's just Openrouter AI/API and your auth token here. And then you clear the default auth token for anthropic. The second thing you need to do is edit your quad settings, JSON, which is in dot claude Settings JSON, you can open up again in whatever code editor you want and change your model to the GLM 5.2 string from open Router. So here I've put it in. And so with those two things, any claude code session that I open up will have the OpenRouter API key. It will route all requests through that OpenRouter API key and it will set the model to GLM 5.2. There's a very similar process that you follow for codecs, but the TLDR of setting up your cursor claude code codecs with a new open weights model is to find a provider, switch out your API key and route all your model calls to that new model. So now I have cursor and now I have quad code running GLM 5.2. And just to prove it to you, I'm going to pull up cursor right here and you can see GLM 5.2 through the API being used right now. Okay, so I'm going to run through a couple use cases of GLM 5.2 and spoiler alert, I have not done any any of this, so I am not sure how it's going to work and just give you my vibe check on whether or not this is a model that I would generally use in my day to day flow. So the first thing I'm going to do is just see how good it is at exploring an existing code base and telling me a little bit about it. So I'm in the chat PRD code base and I'm just going to say here, this is the chat PRD code base. Please explore it and tell me a little bit about its architecture and the most recent things we have been shipping on this code base. So this is going to go through my code base and we're just going to explore how good it is at independently auditing, reviewing and understanding the structure of a code base. From zero. This is one of the most common tasks that you would do as a software engineer is really getting oriented and it's a good reflection of its ability to run autonomously, its ability to use its context window effectively, and its general sense as a software engineer. It was actually pretty fast and it came back with a pretty good overview. So it is a next app, it's got the full stack, it's got a nice architecture here of what it looks like. It's talked about different integrations we have and what we've been shipping in the last six weeks, which is our chat v2 stability. Absolutely. And then some billing and Lenny promo Stuff that we've been working on as well as security and dependency hygiene. So this is actually pretty correct. It was very fast and very accurate. And so just out the gate, this does not seem like a dumb model, but this is a pretty easy task. Let's make it something that we can visualize a little bit more and see if it does a good job communicating agent to human. And we all know that this is the year of HTML. So I'm going to say turn this into an HTML page that can communicate the overall architecture of the app and give a sense of the upcoming roadmap. You can use whatever components you want to make this look good and communicate to me the end developer the major pieces of the architecture and product strategy. Give me a page to pull up when it's ready to review. So again, this is going to take a little bit of the combination of the reasoning and intelligence of the model and combine it with a preview of the design sense and communication sense and see what we get out of it. Okay, it's creating this HTML page for me. It's told me to approve the HTML. I'm gonna pull it over because it does not look bad, you guys. So this is the chat purity architecture and roadmap review right out the gate. It's like slop adjacent. We have blurple on here. That's that blue, purple, indigo color that we love. But it's not ugly. So let's take a look at the content. Well at the high level it's does look correct. Wow. We've merged almost 3, 500 PRs. We've done a lot of PRs. It's giving me a good overview of the core pieces. Oh, this is really great. This is the anatomy of a chat turn. So how the core piece of our application actually works. Some product pillars which are our chat, our integrations, documents and collaboration as well as billing, which also sounds correct. And then it's given me a list of recently shipped things and then roadmap and direction. This is actually the piece where I'm most impressed. One, this looks real cute. And it got the chat PRD pink, which not all models get. GPT wants to give me these like ugly green and navy colors. Claude wants to give me clawed orange all the time. But look at this. GLM gave me chat PRD pink. I'm very happy. And so it's given me what we're working on in flight. And then let's see what it suggests should be up next for our roadmap integrations partnership and enterprise motion. Cost and performance and then knowledge and retrieval. Spoiler alert. These are actually the things that we're working on and a couple conventions about writing our code which is quite nice. So I don't know guys, this is pretty good. This is the first time I run an open weights model inside Claude code and I have to say I am quite happy about it. But let's take it to the next step. Let's design something real this episode is
- [15:29]
B
brought to you by Mercury Banking, designed to work the way modern software does now with Command, a fundamentally new way to get things done with your bank One thing I've become increasingly allergic to
- [15:41]
A
as a founder is dashboards. Not because dashboards are bad, but because
- [15:45]
B
every workflow starts with opening another tab, finding the right report, and figuring out
- [15:50]
A
what to do with the information.
- [15:52]
B
Mercury Command takes a different approach. Instead of navigating your banking software, you simply tell Mercury what you need. Want to understand your cash position? Ask need to create an invoice or categorize transactions? Just ask because it's built directly into Mercury. Every answer comes from your actual financial data and every action respects the permissions
- [16:14]
A
and approval workflows you've already set up. I've always appreciated Mercury because it feels
- [16:19]
B
like a product company built a banking experience Command feels like the next evolution of that idea. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC insured bank. Banking services provided through Choice Financial Group and Column NA Members fdic so what
- [16:39]
A
I have up here is the Chat PRD website but but specifically our How IAI AI landing page and blog where we put every single episode and a summary of those episodes up on our Chat PRD blog. So this is a pretty highly trafficked part of our site and we redesigned this a couple times using AI. But I want to do it again and there's a specific piece of this page I don't really love which is this header section. And so we're going to have GLM 5.2 just make a pass at redesigning this header section and let's see how it does. So I'm going to say here, let's redesign the header Hero section of the How I AI landing page where all the How I AI blogs are, the part that says AI workflows and stories from the experts through the cursor credit claim. I want to redesign it so it is higher quality design, it is a better call to action to workflows and it helps with anything we need on SEO design. Whatever you like. Looking forward to what you make. I don't know, you guys, it's really embarrassing to prompt, but this is what I'm going to give it. Now I've told it to redesign this hero section. It's going to run it through this new model and we're going to see if its ability to redesign even a small section of the page will give us AI slop or if it will give us something a little higher quality. And the reason I like to test on the chat PRD marketing site is it has an existing design system and there are specific things that we really like to see in chat peer to you design. So this will be a good test to see whether or not it can match to an existing design system versus generating a completely novel design like we saw in the architecture overview. So I'm going to let that run and we're going to see what it looks like when I come back. Okay. It says it has a plan and executed that plan. Let's look at it. You know, I don't hate it. I don't quite love it yet, but it's not, it's not bad. What do I like about it? Well, I like the fact that the AI workflows, as requested, are a much better call to action. It also has this nice hover effect on it. I do like that it put sort of some metadata here and some value propositions on who it's for, how frequently we drop and how many episodes we have. And I do really like this little sidebar widget that makes the listen to the show, the calls to action, to YouTube, Spotify and podcasts, Apple podcasts look a little bit more like a player. I'm not sure what this little square in the corner is. And then I do think this copy here is. It might be what it was before, but looks pretty good. I would just say I don't love fully all these colors in the sidebar player. So I'm going to give it that feedback and say I really like this. Except for the listen to the show sidebar, YouTube, Spotify and Apple podcasts are very bright buttons. They're super overwhelming and they're very wide for the text that's in it. I think this component could look a lot higher quality and a lot better for our specific design system. Can you take another pass? Let's see what it comes up with. But I will say for the speed and for certainly the cost, this does not make me unhappy. And I think we would all question how much intelligence do we need to put towards this specific problem. And as long as the model has good taste, I don't need to be fancy and use the most expensive one. So I would say just first glance, first pass.
- [20:16]
B
GLM 5.2 is pretty good at design
- [20:19]
A
stuff and maybe we should all be switching over to it, especially if you are anchoring in something like a design MD or other design guideline or design system where the model can really anchor on it. I do like this a little bit better. It went with a sort of black call to action, a lot more subtle and a lot smaller, but there's some misbalance between the left and right.
- [20:41]
B
But again, it's pretty fast.
- [20:43]
A
In fact, it's almost as fast as Composer is, which is a model by cursor that I use really frequently. And so I think this is pretty good and I would definitely put GLM 5.2 in the rotation for design work. Let's wrap with a much more complicated use case though, which is a long running autonomous use case. So part of how GLM 5.2 has been advertised is that it is a very agentic model that is capable of handling very long running autonomous tasks and solving those over time. Very similar to the claims about Opus and GPT 5.5 or whatever. And so I gave it a common task that I like to give a lot of my long running models, which is pulling issues and error logs and then making a plan for for fixing those error logs and ultimately shipping those fixes themselves. And so before I started this podcast I started with this specific prompt which is pull the last 72 hours of sentry errors in Vercel error logs and build a prioritized plan of bug fixes based on observed issues. And so this has been running the entire time I've been recording this podcast. Probably about 3045 minutes, even though you all will get a much shorter cut of this episode. And and you can see here it did the thing that most models do which is it built a to do plan. It ran tool calls and MCP calls, it read the output. It actually asked me to auth into vercel so that was great. It ran several vercel calls and now it is putting together a plan in HTML I believe for us to review and decide if these are the priorities for chat purity. So I will let it finish writing that plan. But again this has been running for about 3045 minutes and we will see if it came up with something of high quality. Okay, quick intermediary peek from the reasoning minds. It is really struggling to write typescript so while it can do a Long running task. It is having some typescript errors so we may be sitting here for a while waiting for it to write the plan as opposed to its intelligence on getting the plan. So hold tight, we will be right back and I will give you my opinion whether all this waiting was worth it. Oh my God guys, it really is having trouble writing JavaScript right now. So. So okay, we got over the hurdle of it can write HTML is very good at writing HTML. The HTML and CSS is well designed and looks good. I think it can query tools and look at data very well, but I don't think it can write react which is 98% of what I do with these models. So if this is a failure state we're going to have some trouble. Oh it compiled cleanly. I spoke too soon. I just had to complain to the model gods and we are back. It's going to clean this up and hopefully show me its plan on how to fix all the errors in Chat Purity. Okay, here we are. It's pulled the last 72 hours from both sources and built a prioritized plan in a canvas I can open. Here is my canvas again. It does not look bad. I love that it's for an engineer so they made it in dark mode. It says we have 20th century errors, 5 Vercel log signals, 14 planned fixes and then gave me 2P zeros. Not happy about that. This is something that was not coming through on the signal to noise ratio on some of our sentry issues. So we will fix that and then we need to look at this. I think this is a legacy Google Drive integration, but we will take a look at that as well. It's given me events by volume and then runtime log signals and whether or not they are high severity or low severity and then gave me this beautiful prioritized fixed plan, y'. All. I was really disappointed by its speed in writing React, but this is exactly what I need. This is super helpful. It even looks like it's supposed to be part of Cursor which I really love and I can go through and start to rock through these fixes. It even gave me suggested sequencing. I'm not mad so I spoke too soon. Being disappointed about the performance on this long running task. I actually think it's pretty good and I think I'm going to ship all these fixes and apologies to my Chat Purity users if you ran into these errors. I think a few of them are new. So what's my takeaway with GLM 5.2? It's good I would use it for front end work, I would use it for long running backend tasks and the test that we were testing is how much did it cost me? And if you pull up openrouter and your usage on this API key I spent $3.36 on about 6 million token cash. Rate was pretty good 72%. I spent most of those tokens on that 45 minute long running task in cursor to find all my issues in Century and Vercel and just a few in Claude code. But if you compare this to the cost of an opus or a GPT 5.5 this is a steal. So I think I'm going to keep it running in cursor for a while. I think I'm going to keep it running in cloud code for a while. I'm going to see if it can handle most of my tasks and maybe I'll have to buy some more hardware and start running this stuff locally this has been our first open weights model review here at How I AI. I would love to hear what you think. If you want me to try more models, if you want me to show you how to run more open weight models, or if you just want me to explain to you what is happening in this world of frontier coding models, whether commercial or otherwise, I'm here for you. In the meantime, get back to coding and thanks for joining How I AI. Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify or your favorite podcast app. Please consider leaving us a rating and review which will help others find the show. You can see all our episodes and learn more about the show@howiaipod.com See you next time.