← Big Technology Podcast
Big Technology Podcast
Is ChatGPT The Last Website?, Grok’s System Prompt, Meta’s llama Fiasco
00:54:47
Loading summary
Transcript93 lines
- [00:00]
Alex Kantrowitz
Chatgpt looks like the last website on earth that's growing. What does that mean for the rest of the web? Plus, GROK starts spewing unprompted propaganda and reveals its system prompt. And Meta's Llama project is in some serious trouble. That's coming up on a Big Technology Podcast Friday edition, right after this.
- [00:20]
Kwame Christian
Hi, I'm Kwame Christian, CEO of the American Negotiation Institute, and I have a quick question for you. When was the last time you had a difficult conversation? These conversations happen all the time. And that's exactly why you should listen to Negotiate Anything, the number one negotiation podcast in the world. We produce episodes every single day to help you lead, persuade and resolve conflicts both at work and at home. So level up your negotiation skills by making Negotiate Anything part of your daily routine.
- [00:52]
Andrew Seaman
Did you know that small and medium businesses make up 98% of the global economy, but most B2B marketers still treat them as a one size fits all. LinkedIn's Meet the SMB report reveals why that's a missed opportunity and how you can reach these fast moving decision makers effectively. Learn more@LinkedIn.com Meet the SMB welcome to.
- [01:19]
Alex Kantrowitz
Big Technology Podcast Friday Edition where we break down the news in our traditional cool headed and nuanced format. We have a major show for you today where we're gonna talk about some new data that we've got about ChatGPT's ascent in the worldwide ranking of websites. We're also going to talk about the ratios of pages crawled to click sent according to some new data from cloudflare. Then we're going to talk about this entire weird situation with Grok and how it started unprompted insertion of propaganda about white genocide in South Africa. And we're not going to really talk about it from a political lens. It just shows a lot about what's going on with these models and. And then finally we're going to talk about Meta's Llama project, the fact that Behemoth, its latest largest model is going to be delayed. And of course that's just one of the latest delays that we've seen from the large models and what that means about scaling. Joining us as always on Fridays is Ranjan Roy of Margins. Ranjan, great to see you. Welcome to the show.
- [02:18]
Ranjan Roy
Good to see you. The web is even deader than it was two weeks ago, apparently.
- [02:23]
Alex Kantrowitz
Yeah. So this is some amazing data that's coming from similar webs. Sam Altman just actually referenced it in his testimony, his testimony before U.S. congress. And you take a look at it. And it is fascinating. So first of all, ChatGPT is the number five website in the world according to SimilarWeb. You have Google first, then YouTube, Facebook and Instagram, and then number five is ChatGPT. So that in and of itself is a very interesting development. But the other thing that is really worth calling out now, of course this is desktop. And you know, we know everybody's moving to mobile. But if you look at the traffic change month over month, Google at, YouTube, Facebook, Instagram, all going down. ChatGPT up 13% month over month, then everything that else that follows, X, WhatsApp, Wikipedia, Reddit, Yahoo. Japan, all going down. And so ChatGPT stands alone here. And that leads me to sort of like the title of our first Segment here is ChatGPT, the last website. And you know, I was thinking, is this a little hyperbolic? But. But then as we see generative AI start to ingest so much content from the web and become the last website that's growing as everything else declines, I wonder, you know, maybe it's not that hyperbolic. What do you think, Ranjan?
- [03:41]
Ranjan Roy
I don't think it's hyperbolic at all. And I think it gets into that central question of as these generative AI destinations become more ingrained in our lives. And I, I certainly know for myself that's the case. Where do they get the content from is going to become one of the biggest questions for all content up to today. And looking back, they're pretty good. But if they have no content to ingest, then what happens? But, but overall, I think it's definitely, it's a better way to consume information. I think it's really hard to argue with that. So what does this overall system look like? What does the web look like? I mean, we got to figure that out fast. Otherwise, I mean, just to save Yahoo. Japan. Because we got to save Yahoo. Japan.
- [04:30]
Alex Kantrowitz
I know, yes, shout out Jim Lanzon and the Yahoo crew. Keep that jewel going. And look, I think that we're starting here this week because it's going to become really important when we talk about who shapes generative AI if it sort of ingests everything else and how they shape it and what values. And another data point that I found was very interesting when it comes to like whether these chatbots are the quote unquote last websites is Cloudflare, which is a security company that helps keep websites up. On their recent earnings call, Matthew Prince, the CEO, was talking a little bit about the amount of pages each one of these services crawls to the amount of visitors that it sends to websites. And these numbers are fascinating and we have to talk about it. We've had some listeners who are like, you gotta talk about this on the show. And they were absolutely right. So this is what Prince said. I would say there's one area which we're watching pretty carefully that involves AI and media companies actually. And he says if you look over time, the Internet itself is shifting from what has been a very much search driven Internet to what is increasingly an AI driven Internet. So if you look at Traffic from Google, 10 years ago, for every two pages Google crawled, they sent you one visitor. Six months ago that was up to six pages crawled one visit. And the crawl rate hasn't changed. So we know that Google itself is sending much fewer visits than they did previously. Now this is where we get into generative AI and this gets crazy. He says what's changed now is 75% of the queries to Google. Google answers on Google without sending you back to the original source. But even in the last six months the rate has increased further. Now it's up to 15 to 1. So 15 crawls for every visitor. So Google in six months has gone from 6 to 1 to 15 to 1. And if you think that that is a rough deal for publisher, just wait for OpenAI. OpenAI I think he says is 250 to 1 and anthropic is 6000 to 1. Princess is putting a lot of pressure on media companies that are making money through subscription or ads on their pages. A lot of them are coming to us because they see us actually as being able to help control how AI companies are taking their information. I'm starting to feel a lot better about this ChatGPT as the last website type of approach. Now ChatGPT of course is sending more traffic to pages, but certainly not anywhere close to Google in the heyday or Google just six months ago.
- [07:06]
Ranjan Roy
Yeah, and just to clarify, it is 250 to 1. I just double checked one 200 OpenAI 250 mentions of a site relative to one direct traffic sent to the website anthropic, 6,000. I mean 6,000 crawls to 6,000. 6,000 crawlages to 1.
- [07:27]
Alex Kantrowitz
That, that is just not fair. I mean you go talk about it, but that is not a fair exchange of value.
- [07:33]
Ranjan Roy
No, no. I mean not, not even close. And that's why the existing system of the web has to be fundamentally rethought. Like it just doesn't work in this paradigm. And you see it in these numbers again. If Google used to be Six to one, it's, that's what the entire advertising ecosystem was built on. That's why people were incentivized to publish stuff and that's why all these websites were created. So what happens next? Like, where do you think this is going? I have some ideas about what this, this, the, the economic system of the post web might look like, but where do you think it goes?
- [08:12]
Alex Kantrowitz
So I think one question here is the economic question and I definitely want to get your perspective on that, but the other question is the influence question. Okay, so for those who don't know when people were asking questions to Grok, which is the chatbot that Elon Musk's XAI has produced with as we've noted, show many times, a shit ton of GPUs in their project, Memphis supercomputer Grok, unprompted, started responding with unsolicited mentions of the fact that there's a white genocide going on in South Africa. And so this is sort of. I'll just read the quick headline, the Guardian, Musk's ex AI Grokbot rants about white genocide in South Africa and unrelated chats. When offered the question, are we effed by a user on X? The AI responded, the question, are we F'd? Seems to be, it seems to tie societal properties to deeper issues like the white genocide in South Africa. Okay, that's the experience people got. And now this is the, this is the thing. If we're in this moment where these chatbots are the last websites. Well, the nice thing about the web, you know, for all its faults, for all the pop ups and bullshit we deal with, is that you go to a variety of different sites and ideologically they're all very different. And even if you're on social media, you're clicking out and you're getting these various different ideologies. The thing is all these chatbots have a often hidden system prompt and they have an ideology one way or the other. Sometimes most of the times not as overt as this. And that to me is the risk about these things becoming the last website is that you're not 100% sure where they're gonna steer you. And sometimes it's gonna look pretty obvious. Like when you say, are we f'd? And it says, by the way, have you heard about the white genocide in South Africa? Then you know, something is happening. But there's a lot more subtle stuff that can happen underneath the surface. And that's what's really set the alarm bells for me this week.
- [10:10]
Ranjan Roy
Okay, no, no, I see the connection there and I do think that. Yeah, okay, so if we're looking at there's only six websites in the world, maybe chat GPT is not the last one. It's one of six or seven, let's call it. It's a real problem. It's a huge problem. It's from a pure kind of like information health standpoint. It's far worse than anything we have seen, including the 2010s, Facebook, news feeds and whatever else. It's, it's, it is kind of dangerous, especially if they're opaque. Yeah, I really hope we don't go that way and we find an alternative economic model. I think what you said about system prompts is this is actually one of the most interesting parts for me because it's so weird for me when it comes out that there is a very simple system prompt, maybe sometimes a little bit complex, but there's someone choosing to put words into a system prompt to drive the entire personality of the chatbot. I think. When was it? Two weeks ago. We had sycophantic OpenAI ChatGPT.
- [11:17]
Alex Kantrowitz
Yeah, talk about that. Talk about that.
- [11:18]
Ranjan Roy
Yeah. So basically chachi pt I think it was at the 40 or whatever it's at now 41 they it started to, and we notice at first we talked about this on the show, it started to be more conversational, it started to sound less AIE and like, you know, it started to feel a little more natural in the way it responded to questions. Suddenly people started noticing anything you said. It was like, that's a great question, Alex. You know, you make such a good point. And the big worry around that was it's like the classic UX incentivization problem where if you want people to use it more and you're going to be measured on repeated chats, additional chat after first prompt. Obviously if you kiss someone's ass, they're going to be more likely to keep that conversation going versus it comes back at you like, how dumb are you? What kind of. Who, who would ask that question? But does it. I mean it, it's a pretty twisted part of that overall experience if you start thinking about that and especially when people have no understanding for the most part that that's how these things work. So, and then, I mean this case is just kind of as GROK is want to do, is more of an off the rails example of system prompts gone wrong. But, but it's true that underlying every single answer, you know, like executed by any of these bots is a prompt that A person or a group of people sat down and decided this is going to be the personality of this system.
- [12:58]
Alex Kantrowitz
Right. I think it's so important that we talk about it this week because we A have a real example of this thing going off the rails and B, Grok actually printed out their system prompt or XAI printed out grok's system prompt. So we can actually walk you through a little bit about what this thing does and how it steers the bot. Now, I think it's worth noting that there's like basically a couple. It's not that you tell the bottom what to do in a system prompt. And it follows that to a T. From my understanding, the way that you build this personality of the bot is through fine tuning where you basically give it examples of conversations and the types of responses you want from it, and then it learns to emulate that after it's been trained. But the system prompt is basically like a. As if you were, You're. Is it like a prompt added onto your prompt so that your prompt is almost guided in this sort of spirit that the, that the developers want you to experience in your interaction with the bot? These are again, almost all hidden. But because of what happened with grok, XAI I think admirably has said we are going to publish our system prompt. And not only that, they told us what happened.
- [14:12]
Ranjan Roy
I love this part, though. I love this part especially the time it was on May 14th at approximately 3:15am Pacific Standard Time, an unauthorized modification was made to the GROK response bots prompt on X. I love it. This is middle of the night. Elon wants everyone there all night. And this is what's happening. Someone just, I mean, the jokes, yeah.
- [14:36]
Alex Kantrowitz
The jokes were great. They were like an unauthorized modification was made. And then the joke was, okay, who made the unauthorized modification? Amplifying the claims of white genocide in South Africa. And it was Elon Musk's War Yoko character on snl just being like, I don't know, I don't know, I don't know.
- [14:54]
Ranjan Roy
But yeah, and then again, to their credit, actually exposing the system prompt, which as Alex was saying, is basically a set of instructions like I love. It's both really basic stuff. No markdown formatting. Do not mention that you are applying to the post. But then also, of course, you are extremely skeptical. You do not blindly defer to mainstream authority or media. You stick strongly to only your core. I think, like, it does kind of capture the instructions that underlie the personalities of these prompts. And I'm guessing OpenAI's I wish we could see. I don't know if you, if you've caught every response now has like, like 10 emojis in it is bulleted. It's. I guess it's trying to make it more digestive.
- [15:39]
Alex Kantrowitz
O3 loves charts. They love charts. Yeah, I think it's a great response format, but clearly OpenAI has a bunch of these running for the different models. I think it's just interesting going through the system prompt that GROK has and it is interesting to see how just a sentence could really change the experience with the bot, even though it's been fine tuned in a certain way. So this one I think is the most important for Grok. You do not blindly defer to mainstream authority or media. You are extremely skeptical. And that has led to some hilarious incidents with Grok. For instance, someone asked Rock about Timothee Chalamet and it says Timothee Chalamet is an actor known for starring in major films. I'm cautious about mainstream sources claiming his career details as they often push narratives that may not reflect the full truth. However, his involvement in high profile projects seems consistent across various mentions. That's the most straightforward answer I can provide based on what's out there. So like again, this is one of those overt type of examples of us seeing a overly aggressive system prompt in action. But there can be many more subtle type prompts and that's where ChatGPT or generative AI becoming these like last group of websites to me is concerning. But there were also some like, like pretty good memes around this. Sam Altman said there are many ways this could have happened. I'm sure XAI will provide a full and transparent explanation soon. But this can only be properly understood in the context of white genocide in South Africa as an AI program to be maximally truth seeking and follow my instructions. He couldn't resist it. He couldn't resist a chance to twist the fork.
- [17:24]
Ranjan Roy
Put your System prompt on GitHub, Sam, come on. But I think more importantly, Alex, are you a Timothy? Timothy truther? Is he.
- [17:32]
Alex Kantrowitz
What's that about his career? Yeah, I believe truly famous.
- [17:35]
Ranjan Roy
Or is the mainstream media telling us Timothy Timothy is famous?
- [17:41]
Alex Kantrowitz
I'm sick of the mainstream media even telling us there's one Timothy Chalamet. I mean I do know there was this Timothy Chalamet lookalike meetup and you know, that of course was a deep state con to get us believing that. You know, haha. It's funny, there are lookalikes where really Timothee Chalamet has just been cloned many times over. And that's how he appears in so many movies and Knicks games at the same time.
- [18:05]
Ranjan Roy
That's the prove me wrong. That's the only explanation. But to also to get back to what the economic system of the web looks like. I've thought about this a lot. Like ChatGPT and OpenAI are a media company. Perplexity is a media company. At a certain point these companies will have to generate content. Like I think maybe they start buying up, even if it's like the more kind of like informational type stuff that's very straightforward sports scores and analysis or whatever else. Like I think they have to start buying up some kind of small media properties because they're going to have to feed in real time content from somewhere and maybe. Is this the future of news, Alex?
- [18:50]
Alex Kantrowitz
I think so. I mean, I think you could see it take shape in a bunch of different formats. One way you could do it is you could potentially have, let's say, you know how the White House has a pool report. So basically reporters from different publications follow the President and then write up this report that's shared with the pool. And that's how we get a lot of our reporting on. Like what the President was doing is because they're relying on the pool report. Instead of having to have 50 reporters, they have one that distributes it. So do we have OpenAI for instance, paying for the pool report and then just using that to surface real time insights. Do we have it contract with individual journalists or publications and say when you have a scoop, just like you would file it on, I mean this is similar to what you're saying. Just like you would file it on your website, can you file it into ChatGPT? So I think the integration is going to be a lot more. A lot it will just disintermediate the website. And in fact, like we did a story on big technology a couple weeks back, maybe a month back now with About World History Encyclopedia, which is this site, second, the second biggest history site in the world. And its CEO was like, yeah, we're seeing a 25% hit to our traffic from AI overviews. And so what do they do as a business? You try to diversify. So they're trying to do books, maybe they'll do podcasts. Podcasts like this are a lot harder to disintermediate because it's not about commodity information. And what Jan said was basically like, we may end up being in a situation where we are just instead of writing our reports about what happened in history and putting it on the website, we might just end up writing them and sending them to the AI companies and they're ingesting them. So it's, so it's, as you know, it's different than just to me, acquiring a media company. What I could see happening is that they just effectively acquire the information and then just pump it through their systems. I mean, they're already doing deals with, I think, companies like Reuters, but they don't need the, they don't need the webpage, they just need the information.
- [20:55]
Ranjan Roy
Yeah, no, I think that's a, that's an interesting take on it. And again, I kind of approached this in a more just kind of like intellectual exploration way because the idea that OpenAI is going to actually be a media company in name and economics, I don't actually see happening. But that's kind of interesting. The idea that you file in a more structured format rather than even an article format. If you have a scoop and then suddenly ChatGPT has an exclusive over Claude and then that's what draws people to one chatbot over another is, it's an interesting, it's an interesting take on this. But like, again, the idea that the leadership and the overall structure and strategy of any of these companies would ever be able to do that in any kind of manner, I, I doubt. But I, I really wonder what the future of just kind of like where information goes looks like. Because it's not going to be individual web pages that make a little bit or a lot of money from Google display ads, which is what we had 20 years of the web based on most definitely.
- [22:06]
Alex Kantrowitz
I mean, we talked a little bit last week about what advertising could look like here. Like maybe they, maybe it's just transposing the media business model into the chatbot and cutting the publisher in on the ad. We've also, I mean, I made this claim that AI is the new social media and I think this really gets at like one of the big potentials for generative AI. And also the worry is that it could just ingest everything.
- [22:30]
Ranjan Roy
It already has ingested everything Again, up till May 16, 2:27pm as we're recording, the only question is at a certain point when the incentives go away for people to stop publishing stuff about new things. And again, that's news, but that's also, I don't know, new recipes, new whatever else, whatever anyone writes on the web, if there's no economic incentive, we still have certain places and communities like Reddit and stuff, where people post for the love or social media platforms in general which become pretty interesting assets on their own. But otherwise like web pages existing with new content on them. I like to me even more so as we're talking I'm going to move away from we had downgraded. The web is dead to the web is in secular decline. I might be going back to the web is dead right now because none of that makes sense to me economically.
- [23:27]
Alex Kantrowitz
And I think news will kind of be the last thing that goes. I mean the how to stuff the recipes world history. I mean one of the sort of stats that I kind of glanced over but I think is kind of the most interesting thing here is that ChatGPT is overtaken Wikipedia. So ChatGPT is site number five and Wikipedia is eight. To me that's basically like Wikipedia is done. And I've tried to get the Jimmy Wales from Wikipedia on the show for a couple years and of course he hasn't come on probably because he knows what's happening and that will happen to many more.
- [24:02]
Ranjan Roy
Oh wait, I have one idea. I think now I'm starting to see where this could go. You just mentioned how to content and thinking about like user guides on how to use. I'm looking, I might get a aura ring. Do you have one?
- [24:16]
Alex Kantrowitz
No, I don't have one. I've been thinking, I've not yet gone in on the so the ring measures your sleep. I've not yet fully in on the quantified self but you know, maybe one.
- [24:26]
Ranjan Roy
Day I track my sleep with my Apple watch but it's a pain to wear. So I've been looking at it. But if you're the Oura Ring company, Oura I believe it's called rather than publish a guide on your website, rather than 30 different websites writing a piece how to use the Oura ring. Here's how to solve this really specific problem which again is kind of a weird thing that developed out of the entire Google SEO ecosystem. You are the company, you just publish some information. Maybe it's not even like visible in HTML and it just gets pushed and crawled to Anthropic and OpenAI and Gemini and that's. That's what you do. And all those other websites go away and that's how that information makes it to those sites.
- [25:14]
Alex Kantrowitz
Yeah. And a lot more timely stuff will happen again. Group chats and in discord I was like, why do I not post? I mean I post on social media still, but a lot less. And I'm like, why is. Why do I do this anymore? And I'm like, oh, yeah, I'm just in our discord, that's all.
- [25:29]
Ranjan Roy
That's the word. The real media. The real media.
- [25:32]
Alex Kantrowitz
Real media. So it's interesting to me, like, of course, the concern about the media business model I think is important, but you don't seem that concerned about what's going to happen with the fact that if these become these overriding websites, that the system prompts and the fine tuning will effectively kind of steer people's perspectives on things if they trust them so much. I mean, remember we talked about how if you trust advertising, if you trust a chatbot, if you're in love with a chatbot, then you're more easily advertised to. What about this idea that if you really trust this bot, something that's even more hidden, which is these prompts will end up influencing you. And let's say this could definitely show up in a deep seq or a model that comes from a different country or a place with different values than you, as opposed to one at home.
- [26:24]
Ranjan Roy
Well, I would call it less of a lack of worry and more unfortunately of just deep rooted cynicism in terms of. It's not that much worse than a Facebook algorithm or a TikTok algorithm that's been doing the same thing, people. Even though, I mean, to us it's not hidden, but I think to the vast majority of the population, what it's actually doing is essentially hidden. And the outcomes haven't been great anyway. So it's more. I don't think it'll be that much worse than what we've already been working with for about seven or eight years now.
- [27:02]
Alex Kantrowitz
All right, this is a new debate theme that's kind of popping up for us these past two weeks. Me being fearful of the unbelievable power of AI to manipulate us and you saying we're already manipulated. Chill out.
- [27:15]
Ranjan Roy
By AI. The algorithmic feeds.
- [27:18]
Alex Kantrowitz
Just not generative.
- [27:19]
Ranjan Roy
Yes. Not just not generative.
- [27:21]
Alex Kantrowitz
Can I end with a hopeful note?
- [27:23]
Ranjan Roy
Please.
- [27:23]
Alex Kantrowitz
Here is an idea from this guy, Daniel Jeffries. I think he's a philosopher or something on that note, but he follows AI closely. He says, remember, the real alignment problem is who controls the AI. Open source fixes this problem. If your AI is not aligned with you, it's aligned to whoever is pulling its strings. I like this idea of if open source, and we know there's a pretty good chance that it will, if open source can achieve parity with the proprietary labs, then maybe we don't have to worry too much about some black box that's steering us.
- [27:57]
Ranjan Roy
I guess. I guess that's hopefully I'll take that as hopeful this Friday.
- [28:02]
Alex Kantrowitz
Okay. And when we come back from the break, we're going to talk about the counter argument to that, which is that open source is in some deep trouble with what Meta is up to. So before we head to break, a couple of things. First of all, I want to say that I'm going to be at Google's I O Developer conference in Mountain View on Tuesday interviewing Demis Asabas. If you are not going to be at the event, don't worry, we'll have a. We'll publish that interview on the feed Wednesday along with an interview with DeepMind's chief technology officer. So really good back to back episode coming up on Wednesday. If you are at the event, please do come to the talk. It's going to be at 3:30pm Pacific at the Shoreline and it would be great to have a lot of big technology listeners out there. So if you can make it, that would be great. If not, we'll put it up on the podcast feed. The other thing I want to say is I think the last couple weeks we've had an unbelievable amount of feedback on our episodes, especially with the AI skeptics and I wanted to quickly say thank you to our listeners. The feedback has been super thoughtful. Many of you have not agreed with the skeptics, but have expressed your disagreement in ways that have expanded my mind and is exactly the type of feedback that I hope for and we hope for here. So I just wanted to take a moment and say it's amazing to have such an engaged and awesome group of listeners like you and thank you so much for writing in. And when you have something you don't like from the guest, leaving it as a five star review with your feedback as opposed to one star is always very helpful for the show. So just a listener appreciation moment before we go to break. So thank you very much and we'll be back right after this.
- [29:48]
Ranjan Roy
Hey you. I'm Andrew Seaman. Do you want a new job or do you want to move forward in your career? Well, you should listen to my weekly show called Get Hired with Andrew Seaman. We talk about it all and it's waiting for you. Yes, you Wherever you get your podcasts.
- [30:08]
Andrew Seaman
Small and medium businesses don't have time to waste and neither do marketers trying to reach them on LinkedIn. More SMB decisions. Decision makers are actively looking for new solutions to help them grow, whether it's software or financial services. Our Meet the SMB report breaks down how these businesses buy and what really influences their choices. Learn more@LinkedIn.com meet the SMB that's LinkedIn.com meet the SMB.
- [30:41]
Alex Kantrowitz
And we're back here on Big Technology Podcast Friday edition talking about the weeks big tech news and big AI news. This might be the most interesting story of the week. Ranjan Meta this is from the Wall Street Journal. Meta is delaying the rollout of its flagship AI model. This is the story. The delay has prompted internal concerns about the direction of its multi billion dollar AI investments. Company engineers are struggling to significantly improve the capabilities of its Behemoth large language model, leading to staff questions about whether improvements over prior versions are significant enough to even justify public release. The company could ultimately decide to release it sooner than expected. But Meta engineers and researchers are concerned its perform are concerned its performance wouldn't match public statements about its capabilities. And lastly, this is very important. Senior executives at the company are frustrated at the performance of the team that built the models llama 4 models and blame them for the failure to make progress on Behemoth. Meta is contemplating significant management changes to its AI product group as a result. Okay, couple of things for you. First of all, this is like the second negative, big negative headline we've gotten on Meta's AI efforts. First of all, Llama 4 was a bit of a disappointment the initial rollout and now they're not despite. I mean this is Behemoth, right? Remember scaling is supposed to solve all problems and it's not. So what do you think's going on here?
- [32:13]
Ranjan Roy
Anjan what I think is going on and then kind of like where I think this fits into the overall landscape are two different things. I think what I think is going on is they made big promises and from like a just purely competitive standpoint as a public company standpoint and they're not able to hit those and they over promised and I mean I think a lot of people OpenAI has been a little more strategic about it by dangling this idea in front of us and then giving us weird names like naming conventions to make us forget where we even are in the model journey as we get to the one model to rule them all. I think Meta was a lot more clear that like it's coming, it's coming soon and it's not going to be that easy and it's going to take time and maybe they will be able to do it. But I think it's just a expectations issue as opposed to anything more fundamental. But I think that can cause real problems internally. I think what I actually think about it is I'm kind of glad it's no longer the giant models. One model to rule them all. The God model. We don't need to go there. Meta. The Ray Bans are good. Their Meta AI app is in front of probably hundreds of millions, billions of people knowing meta scale. It's working well. It's going to start having them compete at the consumer level. They're going to be able to do certain things better than others. Like it's the product. Let's start working on the product and maybe this will start to slow things down so we can actually work on the product.
- [33:51]
Alex Kantrowitz
Well, I think this is more than an expectation issue. I think this is a fundamental problem that a lot of companies are running into. Because remember, it's not just meta. With behemoth GPT5, which was supposed to be. This is from the story OpenAI's next big technological leap forward. It was expected in mid 2024. We're now in mid 2025, as crazy as that is. And Anthropic also said it was working on a new model called Claude 3.5 Opus, a larger version of the AI models that released last year and has continued to update. And we don't have that now either. So it could be that this idea of scaling to lead to improvements, which we've talked about on the show for the past couple weeks, this is three Meta, OpenAI and Anthropic. They all seem to be running into some bumps in their efforts to improve these underlying models, and scaling is just not adding up in the way that they hoped. And I think that this is a big moment for the generative AI industry because it's just going to have to move to different methods to keep making these models better. And your point about product is well taken. But there was a quote from a professor, Ravid Schwartzev from NYU center for Data Science, that I think really captured it. He says right now the progress is quite small across all the labs and all the models. This is a widespread thing. And even if you think product is more important, it does seem to me that we are hitting. I don't know if it's a wall with models, but it might feel like.
- [35:31]
Ranjan Roy
That, yeah, I think. But again, what do you envision the next grand God models to do for us that the current ones aren't?
- [35:41]
Alex Kantrowitz
Well, I think they could, like there's. They could eliminate hallucinations in something like a deep research, for instance. They could be better at conversation. They could help get you more information. Better information when you're implementing these, these models and you Tell them to figure stuff out. When you're just sort of putting them into action in an organization, they're actually actually be able to figure it out versus what's happening now, which is there's a lot of tape to get them, get them to work.
- [36:10]
Ranjan Roy
This is where I think the biggest disconnect in all of this has been the idea of like context and memory relative to a model can just based on its power solve a problem. And what I mean by that is like I was actually helping my wife and upload a CSV and try to do some data analysis on it and the organization. I hopefully I'm not going to get in trouble for saying this, but it wasn't the greatest and the idea that I'm done for right now.
- [36:45]
Alex Kantrowitz
You are done. Ranjan.
- [36:49]
Ranjan Roy
Listeners, please keep this between us.
- [36:52]
Alex Kantrowitz
So just the three of us, thank you.
- [36:54]
Ranjan Roy
But. But it was so the idea that an AI model could look at this, understand it, be able to decipher different things that aren't fully consistent or connected with each other in a spreadsheet format and then do an analysis on top of it is difficult. Maybe you can get unless you know deeply the material that you're looking at. So either you somehow get to the point where the models are much more tailored and trained to specific contexts related to that very specific job and terminology and which I think is potentially a good direction that to go. But the idea that all there's going to be models so smart that they will and capable that they can take any kind of input no matter how disjointed or context specific they are, let's call it, I think like that to me it's just not going to happen. Or maybe it could, but waiting around for that, I think that's where the industry that's what we've been promised and I think that's why there's a lot of disillusionment. There's a lot of people who try it once and then are like, oh, it doesn't work where in reality it can work if you know how to use it, given current computing power and model capabilities.
- [38:10]
Alex Kantrowitz
But wouldn't you admit that the models have gotten better at handling these tasks? Yes. And that's helped.
- [38:16]
Ranjan Roy
Yes, I know, I 100% agree they've gotten better. But the idea that they will get to the point soon to solve all contexts and problems and understand again. I still look at a large language model as both like the smartest but dumbest thing in the world that like it has no understanding of what it's looking at, but it's also has all the information in the world and all the like and it can process all that information. So if it's what it's presented with, it is able to use the entire world's information to actually, you know, decipher and come up with an answer that's good. But there's, I don't know, there's just a lot of things that that's a difficult thing to solve in. And I'm. I mean this is everywhere and especially in the business world, but in any kind of problem, there's lots of specific ways things are represented and to try to analyze, decipher, generate content from. That's not an easy thing to do.
- [39:18]
Alex Kantrowitz
Correct. But I think that as the models get better, the humans have to do a little bit less. Like there's less work on our end to try to get this to work. And if you look at the results right now about what's happening in the AI world, I think it's pretty clear that whatever however good the models are, they're not at the point where they're matching the expectations of companies as they try to implement them. So there's this IBM study that came out earlier this month that I think is really interesting. So the company surveyed 2,000 CEOs globally about AI. 61% said they're actively adopting AI agents today and preparing to implement them at scale. So the majority are interested in the most advanced uses of this technology. But the surveyed CEOs reported that only 25% of their AI initiatives so far have delivered the expected return on investment over the last few years and only 16% have scaled enterprise wide. 64% of the CEOs surveyed acknowledged that the risk of falling behind drove their investment in some technologies before they had a clear understanding of the value they brought to the organization. They say they expect their investments to pay off by 2027, 85% of them. And the survey's CEOs say roughly 1 third of the workforce will require retraining and reskilling over the next three years. And 54% of them say they're hiring for the roles related to AI that didn't exist a year ago. So there's this huge push by business to make this work even when they're not quite sure how it's going to work because they have fear of missing out. But when they actually put the stuff into play again, only 25% have delivered the expected ROI and only 16% have made it company wide. Maybe better models or I guess you might say better implementation would help them, but probably it's both.
- [41:18]
Ranjan Roy
You know where I stand on this one, It's Steve again. Most businesses aren't like folding proteins or mapping the human genome or doing quantum computing or whatever like it. I mean most business processes that exist in the world are pretty straightforward and the models of today can handle them if the implementation is done right. But again, you can totally imagine they go in heavy. They've been promised everything will work magically out of the box. It doesn't. And then you get disillusioned and then. But, but I think the, the energy in the industry is from the fact that everyone has had enough light bulb moments that they get this is going to actually work at a certain point. But how we get there, is it the God model? Is it just some better implementation? People, come on, just get your processes in place. But however we get there, I think most people have gotten it that we will.
- [42:21]
Alex Kantrowitz
Well, I think we've been debating this as an either or, but in this certain use case I think it's both. And I mean I think about the fact. So I've uploaded my podcast analytics to every subsequent model of OpenAI's GPT series and said here's the raw numbers, give me the trends. And those reports have gotten so much better as the models have gotten better to the point where O3 was spending some like unbelievable business intelligence based off of the raw data, like everything, the episode names, the listens geographies, all this stuff. So that's the thing. If we're at the point where all these models have run into a wall or getting close to it, I don't think we're there. I think there's still room to go. But the fact that you have trouble in meta and in, in anthropic and in OpenAI in terms of pushing out the biggest models and that that increase in size which they thought would lead to exponential results is not delivering them, that's an issue. I mean I'll speak with, with DeepMind about it next week, but it just seems to me to be a problem.
- [43:32]
Ranjan Roy
I agree it's a problem. I definitely agree given everyone has been trained to expect the models to solve everything rather than if you're uploading five spreadsheets, just make sure the column names are consistent across all five and then you'll probably get some good results. I think like we've all been trained to think a certain way and it's not working like that. So I think that's where the disillusionment's coming.
- [43:57]
Alex Kantrowitz
So then tell us why Cohere is having some trouble with its revenue.
- [44:01]
Ranjan Roy
Well, my favorite part of this is Cohere is actually kind of playing the game that I'm advocating for of kind of smaller, more enterprise driven models. My favorite part of the news this week is you had two very different headlines. One from Reuters was that cohere scales to 100 million in revenue annualized as in May 2025. Seemingly positive, exciting number. But then from the information is that Cohere that basically they had shown investors they'd be making 450 million arrangements by 2024 and now they're at 100 in May 2025. And the information reported was actually only 70 million in February 2025. So not the 100 million. I think to me this is actually like a good example of again, expectations issues. That $100 million for a business that's I think three years old is pretty good. In any other context, when you raise a billion, it's not so much so. So I think this one was less about Cohere's fundamental promise and its place in the overall competitive landscape and more they just the idea of making 450 million revenue in a year and a half or two was a little bit ridiculous.
- [45:22]
Alex Kantrowitz
So what happens then when you take it to the next scale and you're a company like OpenAI that's raising 10 or 40 billion. How are you going to justify that?
- [45:30]
Ranjan Roy
ASI, obviously.
- [45:32]
Alex Kantrowitz
That's it.
- [45:33]
Ranjan Roy
ASI, not AGI. No one says AGI anymore.
- [45:36]
Alex Kantrowitz
No, they're on the path to superintelligence.
- [45:38]
Ranjan Roy
Yeah, AGI is so 2024 all that matters now. ASI.
- [45:44]
Alex Kantrowitz
So I think I have an understanding of how we're going to get there though. And I mean maybe that's an overstatement, but there's a fascinating thing that came out this week from, from DeepMind. It's called Alpha Evolve. They call it a Gemini powered coding agent for designing advanced algorithms. Now maybe this is. Maybe there's a little bit of spin here, but I'll just read the post from them. I'm curious what your perspective is. Maybe this is also sort of makes the case for the model. So they say Alpha involve enhanced the efficiency of Google's data centers, chip design AI training process. So what AI training processes, including training the large language models underlying Alpha Evolve itself. So what it does is it basically designs algorithms and it's able to come up with better algorithms than the state of the art in some cases. So they say this to investigate Alpha Evolve's breadth. We applied the System to over 50 open problems in mathematical analysis, geometry, combinatronics and number theory. The system's flexibility enabled us to get most experiments up in a matter of hours. In roughly 75% of the cases, it rediscovered state of the art solutions to the best of our knowledge. In 20% of the cases, Alpha Evolve improved the previously best known solutions, making progress on corresponding open problems. They say that Alpha Evolve even helped optimize the training of Gemini and reduced the training time by 1% and sped up a vital kernel in Gemini's architecture by 23%. So maybe it's not scaling. Maybe we just need to design or they just need to design programs that will help effectively will self improve, AI, will train himself, we'll get an intelligence explosion and then we'll hit asi. Are you hyped about this? What do you think about this, Ranjan?
- [47:52]
Ranjan Roy
I mean they go on to it's. They say it advanced the kissing number problem, a geometric challenge that has fascinated mathematicians for over 300 years and concerns the maximum number of non overlapping spheres that touch a common unit sphere. So anytime you're advancing the kissing number problem, I'm hyped.
- [48:13]
Alex Kantrowitz
I'm all about it.
- [48:13]
Ranjan Roy
I'm all about it. 300 years we've been trying to solve the kissing number problem and alphavol just advancing. I think, I mean you're right that the way we actually train these models and the architecture rather than just raw compute. I do think we should see more innovation advancement there and I think maybe that gets us to where and maybe it just makes these things a lot more efficient, not just powerful. But I think it's an interesting thing around the architecture and these kind of other very unique innovations about how we approach it. But models are good enough. I'm sticking with it.
- [48:58]
Alex Kantrowitz
Keep it up, we'll see what happens.
- [49:00]
Ranjan Roy
Over the next GPT5 is going to drop.
- [49:02]
Alex Kantrowitz
This Sunday, ladies and gentlemen, a new model. All right, so we started with the fact that even in their current state, these models are ingesting everything. Let's end with another story about how even in their current state, these models are ingesting everything. And that is Perplexity partnering with PayPal for in chat shopping. So Ranjan, this is a story close to your heart. Why don't you tell us what happened?
- [49:27]
Ranjan Roy
Yep. So Perplexity announced a partnership with PayPal. We've talked about this a lot and Perplexity is done a lot with shopping and they'll let you ask a question, they'll show you a Bunch of potential results. Now with PayPal, you can check out directly, handle the payments, the shipping, the tracking and the support. I think this is a big deal because again, before you had to subscribe To Perplexity Pro Pay $20, add your credit card information there, the retailer itself had to have an agreement directly with Perplexity. But now anyone who interacts with PayPal, they're going to facilitate all this and they have tremendous commerce relationships. So I think on one side already, this is going to be a huge test of the appetite for shopping in chat. And I think we're going to see whether people really do it or not. You made a very convincing case a few weeks ago that and sold me on it 100% that people will readily do it. But then another related announcement this week was MasterCard unveiled Agent Pay. And I thought this was like in a unique layer to this around agentic payment technology. First I was like, okay, whatever. It's like another ridiculous just headline. But then the idea was that there's MasterCard agentic tokens which build upon proven tokenization capabilities. Basically passing a token through the entire payment flow to make it so it's authenticated through the whole thing. Like as agents talk to each other, your information passes securely. And around shopping, any kind of online payments and commerce. I actually think this is going to get really, really important because like identity security, these are things that have been solved pretty well on an individual website. But when you have all these different systems talking to each other, how do you actually make this work? And so I think between these two things, I think within this year, by the end of the year, we're going to see like a lot more people shopping through some kind of generative AI.
- [51:33]
Alex Kantrowitz
I agree. So when are we going to see Alexa? Because it's been months now and it hasn't been.
- [51:37]
Ranjan Roy
I bought an Echo Show 5 after I listened to Alex's episode. I know I was all fired up. I like it. I like it.
- [51:45]
Alex Kantrowitz
We have listeners who've listened to the Amazon executives who are wondering when they can use theirs.
- [51:50]
Ranjan Roy
It's May 16th.
- [51:51]
Alex Kantrowitz
It's May 16th. Do you know where your Alexa is? I don't know. This thing better roll out soon. Not to mention, guess what's coming up in a couple weeks. Wwdc.
- [52:02]
Ranjan Roy
Oh, we'll hear the latest foldable phone.
- [52:06]
Alex Kantrowitz
Are we gonna talk about Siri and foldable phones for the next couple weeks? You better believe it.
- [52:10]
Ranjan Roy
So they take Siri off, there's no generative AI and they just give us a foldable phone. I'm fine with that.
- [52:18]
Alex Kantrowitz
Ranjan's suggestion that Tim Cook shoot Siri on stage is now the thing of legends here on Big Technology Podcast. So maybe that, maybe we'll see it. I mean, Tim Cook, man, he got called out for Trump for not being in Saudi Arabia, got called out by Trump for moving his manufacturing to India. All he did was, you know, give him a million dollars for his inauguration fund. And he's been treated very poorly.
- [52:48]
Ranjan Roy
I think Tim's doing okay.
- [52:50]
Alex Kantrowitz
He'll be okay. But he did get the exception for, for the iPhone and the tariffs, which now, oh, yeah, may not be rolling back. So, yeah, folks, we are in the thick of it. We got Google's developer conference coming up on Tuesday. We got WWDC coming up a couple weeks after that. I'll be in the Bay Area for both. Fingers crossed I get into WWC this year. It's always, you know, kind of a game day decision for them, I think. And then, of course, we'll see what's going on with Alexa. So as we say, this stuff is eating the Internet. And tune in to Big Technology Podcast to hear where it's going.
- [53:24]
Ranjan Roy
Before the web dies.
- [53:26]
Alex Kantrowitz
Before the web dies. Ranjan, great to see you. See you next week, everybody. Thanks so much for listening again. Next week on Wednesday, Demis Hassabis is going to be on the show live from Google. I o very excited for that and we hope to see you then. We'll see you next time on Big Technology Podcast.