Loading summary
Transcript109 lines
- [00:00]
A
Welcome to the AI Chat podcast. Today on the podcast, I want to talk about a really interesting concept that might be throwing a serious wrench into OpenAI, anthropic and a lot of the other big players in the AI market. And that is the fact that right.
- [00:14]
B
Now there seems to be a limit that a lot of these companies are.
- [00:17]
A
Hitting when it comes to how fast they can actually improve their AI models.
- [00:22]
B
And the big thing that people are.
- [00:23]
A
Talking about here specifically is right now we went from, you know, ChatGPT 3 to 5 with a really big jump. People are concerned and inside the company are having some alarm bells that the jump from 4 to what a lot of people are calling ChatGPT 5, which they've actually changed the naming convention probably to avoid this issue a little bit, they're calling it Orion. They're worried that this is not as significant of a jump. There's some people inside the company and they're actually having to turn to other solutions to make these models better because of data and compute constraints. So today on the podcast, we're breaking down exactly what's going on in the AI landscape, where everything sits with some of these issues and how people are trying to get over some of these.
- [01:05]
B
Hurdles or roadblocks in compute.
- [01:07]
A
We're seeing some really bold predictions by Opening Eyes CEO Sam Altman and also from the CEO of Anthropic. So we'll touch on those. But there's some really interesting stuff. Uh, let's get into it. Before we get into it, I wanted to mention, if you haven't already and you're interested in making money from AI tools, I would love to have you.
- [01:27]
B
As a member of the AI Hustle School community.
- [01:30]
A
This is a place where every single.
- [01:31]
B
Week I record an exclusive piece of.
- [01:33]
A
Content that I don't put out anywhere else. And I cover how to use AI tools, how I'm currently making money and growing and scaling my business. Um, there's over 200 members that are from, you know, they've started a hundred million dollar companies and some are starting out brand new companies. So a really wide range of perspectives, but a really incredible community to essentially.
- [01:54]
B
Learn how to grow and scale your business with AI and how to make money with AI tools.
- [01:59]
A
So if you're interested in this, you can check it out. There's a link in the description. It's $19 a month. I'll increase the price in the future, but for now you can lock in that price and it'll never be raised on you. And it's a great community. I'd love to have you as a part of it. All right, let's get into the episode. So a big part of this actually.
- [02:15]
B
Came from an article that was broken by the information.
- [02:17]
A
The information always has kind of these juicy tidbits. They interviewed some people inside the company and essentially found out some interesting information. So the big thing here is the challenge right now. The increase in quality of, you know, this next flagship model, Orion, or what some people call GPT5, is not going.
- [02:37]
B
To be as big as the jump in the last two.
- [02:41]
A
Because of this, the entire industry, not just OpenAI but a lot of other players, including Anthropic, are shifting their efforts to essentially focus on different ways they can improve these AI models. Not just the data that goes into it, not just how much compute, but actually software.
- [02:55]
B
There's even a really pessimistic opinion that's.
- [02:58]
A
Held by Mark Zuckerberg. So it's definitely an interesting time to be in the industry. But Mark Zuckerberg, he said that in a worst case scenario there's, he's like.
- [03:09]
B
Don'T worry, there's still like lots of.
- [03:10]
A
Room to build consumer and enterprise apps, you know, different products on top of the current technology, even if it doesn't improve. Okay, this, I for one hearing this from, you know, one of the companies Meta, who is, you know, one of the top five leaders in AI and is coming out with really big cutting edge open source AI tools. This is not super optimistic news for me. He's like, don't worry guys, like, even if we, you know, don't improve it, there's lots of like, apps we can.
- [03:34]
B
Build that are super useful.
- [03:35]
A
Yes, there's lots of apps that can.
- [03:36]
B
Be built that are super useful.
- [03:37]
A
People are going to make those. But like, we really want the underlying.
- [03:40]
B
Technology to get better.
- [03:41]
A
And it's interesting because at the same time we're having Sam Altman say, you know, we're going to achieve AGI in 2026, I believe, or 2025. So anyways, he's got some really bold.
- [03:54]
B
Predictions and his predictions are essentially based off of the rate of improvement that we're currently seeing.
- [03:58]
A
He's like, look, if you look at, uh, you know, the MO. If you look at the DaVinci model.
- [04:02]
B
To the O3 model, to the O4.
- [04:03]
A
Model, or you know, to GPT4, like.
- [04:06]
B
These are massive jumps in improvement and.
- [04:08]
A
If we can keep this up, we're going to hit AGI by, you know.
- [04:11]
B
Next year or the year after.
- [04:12]
A
And I think that the CEO of Anthropic was a little bit more conservative and said, you know, along the same lines that 2026 or 2027, he expected us to achieve AGI, which, you know, people have different. They have different kind of how they classify that. There's a couple interesting things, though, that.
- [04:32]
B
I think go into this.
- [04:33]
A
A lot of people are really excited and hyped. Sam Altman is the first to say that he believes it's going to take 10,000 days, which I think is, you know, a funny way of saying, but he thinks 10,000 days, we have like, super intelligence. And, you know, you always post on Twitter, he's like, it's crazy to think we're just 10,000 days away from super intelligence. It's all kind of extrapolating from the same thing. It's interesting because it would appear they have a different definition for art. Artificial general intelligence and a super intelligence. Superintelligence is something, you know, they hope to achieve, that it's, you know, knows everything, it can do everything, it can predict everything. It's insane. But what they're focusing on right now is a general intelligence, which is generally smarter than a PhD student and a human and can at all tasks. Right? That's what AGI is.
- [05:14]
B
The reason the definition of the AGI is important and achieving it is important.
- [05:18]
A
For Sam Altman is because as soon as he achieves AGI, he's out of the deal that he has with Microsoft, where they get access to the AI models and they potentially could just clot back. Now, I bet there's going to be a lot of legal battles around how exactly that plays out and if they can actually classify it as AGI. But anyways, that's his incentive for saying he's going to achieve AGI next year or the year after. And if he's saying he's going to be AGI, then the CEO of Anthropic has to also say, hey, look, we're.
- [05:44]
B
Going to achieve AGI too.
- [05:45]
A
Because they don't want to be like.
- [05:46]
B
Oh, we're not going to do it.
- [05:47]
A
For five years and make all of a sudden open. I look like, you know, the winner. So now all of a sudden, everyone's got to say they can achieve it. And I think what's going to happen is essentially we're just going to lower the expectation of what we're calling AGI.
- [05:58]
B
So whatever, achieve in a year is.
- [05:59]
A
Like, yeah, I told you we, we'd reach it. This thing's pretty good. It's like, yeah, maybe it's not perfect, but anyways. That's my, that's my take on why they have those bolder predictions. While it seems like they're, they're struggling. So some researchers at the company, at OpenAI right now believe Orion, this is the latest model, is not reliably better than its previous version, which is GPT4. This is not good news if it's accurate, and that's just for certain tasks. So according to this employee that spoke to the information anonymously, Orion is better at different language tasks. So that's good, but it might not actually outperform the last model that they had when it comes to coding and a number of other tasks. But this is really important. Coding is a big, a big deal because this has a lot of logic.
- [06:46]
B
And reasoning that's needed for this.
- [06:48]
A
So definitely this could be a problem, especially when you consider the fact that.
- [06:53]
B
Orion is going to probably be More expensive for OpenAI to run in its data centers compared to what it currently.
- [06:59]
A
Has on the market. So it's going to be charging more money and maybe it's better at some language things, but it's not better at coding or everything. This is going to be an issue that they're going to have to try to solve. So right now with Orion, this whole situation is really going to, it's going to test the assumption of the whole AI industry, which we're all looking at right now, which is the scaling laws.
- [07:23]
B
And that is that LLMs are going.
- [07:25]
A
To continue to improve at the same.
- [07:26]
B
Pace as long as they have more.
- [07:27]
A
Data to learn from and more computing power to essentially facilitate the training process. And this is kind of what Sam Altman and the CEO of Anthropic have.
- [07:36]
B
Both been saying when they're talking about.
- [07:38]
A
When we're getting AGI. So they're really publicly endorsing this kind of scaling law idea. And so this could be a problem if Orion comes out and it essentially.
- [07:51]
B
Has less improvements, less of a jump.
- [07:54]
A
All of a sudden. Sam Altman's timeline and direction for achieving AGI, it gets like, lopsided. It's like, well, if you get diminishing returns, it's going to take us way longer to achieve something that's actually that good. So in response to all of these different challenges, the AI industry essentially is shifting how they're addressing the scaling law, their efforts. You have Mark Zuckerberg saying that worst case scenario, we can still build cool tools on top of it. OpenAI is actually baking more, you know, co writing capabilities into their model because they're trying to fight off anthropic who's a serious threat. And essentially they're just developing software that can, that can do a lot of what they need. And Anthropic right now is even going so far as creating software that can take over your computer and complete a bunch of tasks for you. So they're trying to make their AI models more useful, but it's not because the AI model itself is necessarily getting smarter. It's just they're adding extra software functionalities that make them more useful to a person, which is fantastic. But you also need these underlying, like we need the models to keep getting better and better because anyone can build the software. But you know, some of these guys, there's only so many people that have enough resources, talent and money to make the underlying models better. That's really what we want them to focus on. So what's interesting is everyone's focusing on AGI, everyone's focusing on agents, which is kind of what the next step that they're all that we're all taking. There's a researcher over at OpenAI who.
- [09:23]
B
Is Noam Brown and he gave a.
- [09:25]
A
TEDX conference last month talking about essentially that these more advanced models might actually become financially unfeasible to develop. So there might come a point when it doesn't even make sense to develop. And this is what he said, quote, after all, are we really going to train models that cost hundreds of billions.
- [09:44]
B
Of dollars or trillions of dollars? At some point the scaling paradigm breaks down.
- [09:48]
A
So this is also interesting, right, because they keep saying, like, hey, look, with more data, more compute, we make these things better and better. But like, at some point is it.
- [09:54]
B
Worth spending a trillion dollars to train the latest model?
- [09:57]
A
Like, are we getting bang for our buck? Is there an actual realistic trade off here? So I think these are all really fascinating questions. I'm sure Nvidia is happy with all of them because a trillion dollars sounds probably fantastic to them, but a lot of that goes into power and other like another cost as well. But you, I would say, expect Nvidia to keep growing and benefiting if this is really where, where the industry goes. Okay, so OpenAI has done a lot of work with their Orion model. A lot of people are testing it but, but right now they have their.
- [10:27]
B
Kind of testing and safety that they're doing right now.
- [10:30]
A
It's quite intensive and they're still going into it and yeah, doing a lot of work on that. So some people also speculate that we.
- [10:39]
B
Are going to be hitting a data wall.
- [10:41]
A
This is something a lot of people Talk about.
- [10:42]
B
So this is one reason that they believe we're going to see a slowdown.
- [10:44]
A
In the progress of these different models is because essentially we're getting less and less access. Like the pool of really high quality.
- [10:53]
B
Text and other data that we can.
- [10:54]
A
Give to these AI models is, you know, dwindling, like we're getting less and less of it. We've used so much of it, there's only so much you can create. Now some people say the solution is synthetic data. There's a bunch of, you know, arguments about that back and forth. So this is, this is an interesting topic, but it's no doubt that new data sets seem to be in short supply at the moment. So in the last few years, LLMs really just used all of the data and text that were on websites, books and anything else that's been put onto the Internet. But yeah, most of that's all been used. So I wanted to talk through just the different steps that an AI model takes. So this is essentially the training process, the testing process that these LLMs go through before they're actually released. Just to give you an idea of like where Orion's at and where some of these other models are at and where we're kind of moving to the future. So the first step is setup. That is the data collection and the data pre processing. So cleaning the data, making sure it's actually usable by these AI models.
- [11:51]
B
Next we go to pre training.
- [11:53]
A
So pre training happens, there is continuous evaluation going on and then we go to the evaluation which is essentially evaluating. After the pre training of that model has done, we then move to post training which is essentially introducing a bunch of new data. After collecting and kind of pre processing.
- [12:11]
B
Happens, you're fine tuning the models.
- [12:13]
A
Then you do reinforcement learning based on kind of human evaluation.
- [12:18]
B
So people are testing the models, coming.
- [12:19]
A
Up with like feedback on things that.
- [12:21]
B
Needs to do better or change.
- [12:22]
A
And that's where you get the fine tuning, the reinforcement learning. You then do a pre release, which is essentially what's happening with the Orion model right now, where you have a pretty decent model, but people have to do safety and that kind of, you know, testing, make sure everything's good. And then we get the launch. So we're getting close on the latest Orion model, right? And so when people are giving their speculations on how good it is, it's.
- [12:44]
B
Already been through all of the these steps.
- [12:46]
A
It's just on the pre release right now. It's just on the safety testing and some of the further evaluations before they actually do the launch. So we're quite advanced in this. Okay, so one thing that I think is really interesting is the fact that we have Ben Horowitz. So one of the CEOs or one.
- [13:04]
B
Of the founders of A16Z, one of.
- [13:06]
A
The biggest venture capital funds, who gave a really interesting quote on YouTube recently. He said, we're increasing the number of graphic processing units used to train AI at the same rate, but we're not.
- [13:16]
B
Getting the intelligent improvements at all out of it.
- [13:19]
A
He didn't really elaborate further on it, but it would appear to be from what he's saying, that they're actually giving it more GPUs, they're giving it more processing power and the intelligence isn't actually improving. So this is. Some people are concerned about this and it's going to be interesting to see what happens. So what is the other solution? We've talked about the solution of, you know, adding new software that can kind of do things. So if it's bad at math, you can add like a math software that it essentially queries and it uses that to solve some problems. It's not really the AI model that's doing it, it's just using a math calculator to do stuff. That's a great solution. What other solutions are we talking about?
- [13:58]
B
The big one that you've all recently seen is how ChatGPT came out with their OH1 preview. Right now they're GPT O1 and essentially.
- [14:06]
A
That is the ability to, when you ask it a question, it runs that question through more compute power. So instead of just having CHAT GPT give it straight up vanilla response, it's going to take that response, run it through and say, okay, test this for accuracy, test this for this. Make sure to fact check this, break this, break this question down that they're.
- [14:26]
B
Asking into seven steps.
- [14:27]
A
Complete the seven steps. Now put those steps, seven steps back together. Now condense it. They're doing all this stuff in the back end where they're essentially have like a really elaborate prompt, really working your question out in a much more elaborate way where they do, you know, the. Essentially they work through a thought process. So this is a fantastic tool that is getting it better, but it costs more money.
- [14:50]
B
So Brown, who is doing that whole.
- [14:52]
A
TEDx AI talk recently was talking about this and said, quote, this opens up a completely new dimension for scaling spending.
- [15:02]
B
A penny per query to $0.10 per query.
- [15:05]
A
So essentially researchers can improve the model responses by just spending more and essentially putting it through more rounds. So Sam Altman also has talked about the importance of OpenAI's reasoning models, which it's kind of like this O1 preview.
- [15:21]
B
Which can be combined with their LLMs.
- [15:23]
A
Meaning it's actually not having to retrain a new model to make it better, they're just able to use the same models to get better results by using more processing. So Sam Altman said, quote, I hope reasoning will unlock a lot of things that we've been waiting years to do. The ability for models like this to, for example, contribute to new science, help write a lot more very difficult code. He was talking to developers back in October and was going over all of this. We're seeing that with more time, the act and being able to essentially run the prompts and the outputs through more AI models and giving them more time and compute actually does increase the accuracy, which is great, but it's not perfect. The one thing that I will say that is great with all of this is just the fact that. It's just the fact that while we might be hitting some sort of plateaus on some of the models, being able to run things like, you know, run the model, run this prompt through 10 different things and make sure to pick.
- [16:23]
B
The best and refine it with each.
- [16:25]
A
One and go through step, you know, through the thought process and different steps like that makes it better. And we essentially can just keep doing that over and over again.
- [16:32]
B
Okay, take this prompt and run it through a hundred times and now it's.
- [16:35]
A
Going to get, you know, marginally better than 10 through 10 times. So while it does get really expensive.
- [16:40]
B
You can get better results from this.
- [16:41]
A
Now this is what we're kind of currently doing to get around some of the limitations we have. But I think at the end of the day, we need to confront the.
- [16:48]
B
Limitations head on and say, how do.
- [16:51]
A
We actually get the foundational models better? That's the big thing without spending a trillion dollars on computer.
- [16:56]
B
A lot of that is making the.
- [16:57]
A
Model training more efficient, getting energy costs down. There's a lot of things that we need to do to essentially unlock this and a lot of roadblocks that we have to get through. But it's a very fascinating time to be an AI and it's a, it's a serious problem that people are grappling with, which is that the AI models appear to be having a slowdown and improvement just based off of more compute and more data. And so we're going to have to.
- [17:21]
B
Come up with solutions for this.
- [17:22]
A
So exciting time to be alive. I don't think that this means, you know, they're permanently stuck in limbo or a plateau, but we're going to have to come up with new creative ways to grow. It's not just the same old same old we've been doing for the last two years, we now need to come up with new solutions. So definitely exciting. I'll keep you up to date on everything going on in the AI industry.
- [17:40]
B
And everything that I'm seeing here.
- [17:42]
A
So again, if you're interested in making money with AI tools, make sure to go check out the AI Hustle School community, an incredible place. And also make sure to go and get on the AI Box waitlist if you're interested in building your own AI tools in using our playground that lets you access tons of different AI models all for one subscription cost. So you can go to AI Box AI links also in the description to get on our waitlist for that which will be launching shortly. Thanks so much for tuning into the podcast today. Hope you all have a fantastic rest of your day and I will catch you next time.