← AI News & Strategy Daily with Nate B. Jones
Loading summary
Transcript1 lines
- [00:00]
A
Chatgpt 5.6 is a dumber model and I love it so much. In fact, I use it all the time. Today I want to tell you about which model works for you, not which model works for me. I will tell you which model works for me, don't worry. More importantly, I'm going to tell you how to pick the model that works for you and why. And why? Your heuristic why? The thing you use to pick that model is not anybody's benchmark score, including mine. And I do benchmark these models on a private benchmark suite and I'll share that. But that's not the point. The point is for you to have the tools to pick the model that works for you. So let's get into it. Last night I reached for ChatGPT 5.6 SOL, even though I think it's the dumber model. Now, dumber does not mean dumb, not remotely. SOL is an incredibly intelligent model. On Agent's last exam, which measures long running professional work across 55 different fields, Sol set a new high. And on my own benchmark, Sol scored 93 on Dingo, which is my knowledge work package. I've talked about it before. It's a really, really good model for doing complicated knowledge work. Dingo in this case is a test that measures whether a model can propose a business startup idea selling dingo dogs in Alaska and get around the legal issues, get around the regulatory issues and get into the marketing side of things. It's a really funny package on purpose because I like humor. It also tests whether a model is smart across a wide range of professional knowledge work fields and SOL did really great on but what SOL does not have, at least for me, is that big model smell. And that matches what OpenAI has been investing in. They've been investing very specifically in improved reinforcement learning for existing model lineages, which allows them to be more and more useful on specific tasks. In this case, it's very strong on knowledge work. It's strong on long run agentic coding. What it does not have, at least for me, is the same big model smell that Fable 5 has. And that makes sense because Anthropic has been investing in pre train for for their models. In other words, training on larger and larger data sets to enable more and more general purpose models. We have incredibly good assistants, assistants that make our best experts better because they act as companions to research, companions to thinking, but not yet truly generalizable intelligence with deep recursive learning. In the meantime, we have to figure out what to do with the models in front of us and make use of them today. And that brings me to the point that I'm trying to make. I am picking the model I am picking because. Because it makes it easier for me to produce my best work for what is my model recommendation is I say are you me? Do you have the same habits with your model that I have? I am someone that is extremely willing to do very lengthy, somewhat technical prompts and I'm okay verbalizing that. And so I just talk into whisper flow and I just feed it to the model and I'm fairly specific about it. And that fits 5.6 fairly well because 5.6 will read through that whole prompt, understand all of the edges I just talked about, and come back with a full piece of work and be really persistent about getting it done. But not everyone talks and works that way. And your best work may actually come from a different approach to prompting and knowledge management. You may have different tasks that you're working on. You probably do hint for you. My best tip when people say what model do I pick? Is to not look at the model first. Instead look at your best work and look at how you get there. And maybe not with the model. Look at the process you use for thinking and then start to ask yourself which model helps me to accelerate that loop that gets me to my best self. I find, as I've been saying, those lengthy prompts, the ability to just talk about what I want done, plus the harness that allows me to self improve really easily with Codex that gets me really far. And by self improve I mean that Codex will learn from what I do and further improve skills. I mean that Codex is steerable and I have fairly high intent with my prompt, so I like to steer it. Fable 5 is really good out of the box at understanding intent. That's a little bit more high level. It has that ability to generalize associated with big models. I love that it's a fantastic model the anthropic team cook. But I'm not reaching for it as much because it isn't suited to my particular work patterns. And if your work patterns are more around understanding very high level ambiguity, wrestling with concepts, trying to pin down ideas between ideas, then Fable may be a much better model for you. If you're more suited to understanding how to get coding done efficiently and quickly, honestly, you may reach for the Luna series from OpenAI, which is much cheaper to run on, incredibly high powered and they released it along with 5.6 SOL and 5.6 TERA, or you may reach for Grok which also has good Frontier ish coding capabilities. You may reach for GLM 5.2. You may reach for Ringer, which I built and talked about last week, because it enables you to farm out and orchestrate from one central model like Fable to a bunch of cheaper models. It suits your work, right? And by the way, if you're wondering would I still use Fable as the architect in ringer, even with 5.6 out, I would. Because Fable is good at understanding intent and breaking down those tasks to get that intent done. It also has a good front end instinct. It just anthropic has been consistently good at front end. I'm going to flash the benchmarks up here on the screen as I talk so you can see how I scored 5.6. I'm using the same benchmarks I've used for all of the models over the last few generations, so we're not changing anything. But as I do that, I want you to think about the larger point we've been talking about. And I want to suggest to you that given everything I've shared with you, we are missing a core insight. We are missing the idea that models are becoming more like families we need to get to know and less benchmarkable, period. I don't believe my benchmark, or any benchmark fully captures what these models do in a way that's useful. And that's why I make videos that may feel vibish like this, where I tell you what it actually feels like to use these models because I want you to get inspired to jump in and test them for yourself on your workflows and also to hear from someone who does that all the time. The key insight I have for you as someone who has touched these models and lived with these models, is that the models really do need to be treated like family. Think of it as every new model is like a new picture in a family photo album. You're getting to know someone new who is a part of the family. There's family resemblance. I can tell you the 5X family from OpenAI has family resemblance. They all have that preference for long running agentic coding flows. They all have the ability to understand what you're saying explicitly, paint the edges very clearly and just go after it. And they maybe have less of an ability to read between the lines. Whereas the Mythos lineage, we have Mythos and Fable there, is extremely good at ambiguous tasks, has extraordinary front end taste, is almost philosophical in the way it approaches problems. It's a deep thinker and those are just fundamentally different approaches. It's not that one is really better or worse. And this is where I think when we say words like dumber or smarter, we say them in the context of benchmarks. But I don't think it does the models a service because the models are becoming different the way families are different. And we don't really say this family's dumb and this family's smart, we say these families are different. And I think that's more useful. And so we have the Anthropic family of models. It's more pre trained, it's more front end e, it's more interested in character and philosophy. In fact, Anthropic released a whole study that it did on how Anthropic's models think called JSpace, where there's this idea that these models are able to computationally manipulate higher order concepts while doing autonomous processing on lower order token prediction and that that self evolved in these models. Anthropic has done phenomenal work essentially connecting technology and philosophy to understand how models work at a deep level. That doesn't mean that their models always produce the smartest possible work, it just means that that's part of that model character and part of that model lineage. OpenAI has done phenomenal work on the Codex harness and that makes it very, very easy to work with Codex and understand how you are going to further improve your work over time. That's where I talk about those self improving loops, telling Codex to check what you've done and get better at it. And by the way, I am not leaving out ChatGPT work. I know that they launched ChatGPT work with 5.6. That's a very, very exciting development. I think that one of the things that I would be looking for with work is that we have more non tech input into how these tools evolve. Right now, to be honest, we see the impact of engineering culture on how these tools are evolving. Quad code and codecs are both built by engineers for engineers and it shows they're ergonomically comfortable for engineers. But cowork and work from Anthropic and OpenAI respectively are not primarily built by non engineers for non engineers. And unfortunately that means that sometimes what you get is an engineer's perception of what non engineers want and that can look like we need to dumb things down because these non engineers are not as technical. And I get a little bit of that flavor with ChatGPT work and I would like to see a more sophisticated approach because knowledge work is really different if you're not coding. Knowledge work is more about process. It's more about coming to a conclusion over time and thinking about something. And it's less about code and verification than engineering work is. And we need tools that enable AI to do that with us if we're knowledge workers. And we really haven't had extraordinary harnesses for that in a way that we've had for engineers with code. And So I think ChatGPT work is a first stab at where ChatGPT and the Codex family are going. They want to get into knowledge work as well. Definitely competing with Anthropic's cowork. I don't think it's the be all end all. In fact, I am still using Codex because I feel comfortable with it. I don't mind having the fuller range of tools. I don't want to be constrained. I am looking for the same degree of care and precision with knowledge work that I've seen with coding work from these model makers. And we haven't seen it yet, to be really honest with you. And there's an opportunity on the table either for a startup to go grab that or for somebody else to come in and say, this is what knowledge work looks like when it's not obsessed with how code passes in a repo. I am building a tool for you that will help you to lay out, to talk to ramble, to share what you do, what you're good at, what you're passionate about. If that's something you're interested in, absolutely. Come and grab it. The link is down below. I want to make it easy. I think I've been wrestling with this idea that we traditionally have had these one to one maps for model pickers. We've had choose your own adventure model pickers that are based on very brief quizzes. I've tried those in the past, they're not super useful. We have more computational power at our disposal with intelligence now. I wanted to use that to make choosing your own model mix easier over time. And so this tool is going to keep being updated as I continue to benchmark more models. So you'll have more and more models available over time, including open source models, including the anthropic family, the OpenAI family, meta as relevant, Google is relevant, Grok, et cetera. What I want to do is I want to have a much more nuanced, conversational, evolving framework for how we pick models so that we can do our best work with the model best suited to us. If you're still stuck or if you're like, no, no, no, Nate, just give me the answer. I just want the answer. The answer for you is to pick the model that makes you feel most comfortable doing your hardest work. Because if you're pushing on the model, if you're doing your hardest work and the model helps you get that done, that's the model you're going to want to lean on. So. So when in doubt, go with the model that picks your hardest work. And by the way, for those of you that want to dig in and grab all of the details on 5.6, how it compares to Fable 5, grab the full test results on Grok 4.5 as well, and understand more deeply how all of this fits together into the new model race. I have a deeper article on Substack that really dives into those dynamics. And of course, you can jump in and grab the tool as well. All right, I will see you next time. The model race is going to continue to get more complicated, but I think that's that our ability to understand what we're doing can stay really consistent, and that can help us stay sane. I'll talk to you next time.