The AI Podcast: The Hype vs. Reality of OpenAI Agents

Release Date: August 7, 2025

Introduction to OpenAI's Open Source Models

In this episode of The AI Podcast, the host delves into the recent significant move by OpenAI, which has released two open-source models for the first time in five years, dating back to GPT-2. This development marks a pivotal shift in OpenAI's strategy, drawing both attention and criticism from the AI community and prominent figures like Elon Musk. The discussion begins with an overview of the differences between "open source" and "open," the performance benchmarks of these models, and the broader implications for the AI landscape.

[00:00] "This is actually really big news because this is the first time in five years that they've actually dropped any open source models back to GPT2."

Benchmarks and Performance

Code Force Benchmark

The host explores the Code Force Benchmark results for OpenAI's open-source models, highlighting their performance compared to existing proprietary models.

• 120 Billion Parameter Model: Achieved an Elo score of 2600.
• OpenAI’s O3 Model: Scored 2700.
• OpenAI’s O4 Mini Model: Slightly higher at 2720.

These results indicate that OpenAI’s open-source models perform comparably to their proprietary counterparts, outperforming older models like the O3 Midi, which scored 2000.

[04:30] "These are, these things aren't very far apart. It definitely did better than the O3 Midi model, which only got 2,000."

Importance of Tools in Benchmarking

A critical point discussed is the role of tools in benchmarking AI models. Tools refer to integrations like calculators or specific applications that assist the AI in completing tasks more accurately.

[08:50] "Tools basically mean they gave the AI model things like calculators and apps and like different tools. So like it is completing the tasks. Yes, but it's able to rely on like actual hard software to get good results."

The host emphasizes that while OpenAI's open-source models are released without these proprietary tools, the benchmark scores remain meaningful as developers can build custom tools to enhance model performance.

Humanities Last Exam Benchmark

Another challenging benchmark discussed is the Humanities Last Exam (HLE), designed to test complex, interdisciplinary questions.

• 120 Billion Parameter Model: Scored 19%.
• 20 Billion Parameter Model: Scored 17%.

While these scores may seem modest, they surpass the performance of competing open models from companies like Deep SEQ and Quentin, although they lag behind OpenAI's closed-source models.

[12:15] "I was actually really impressed that the 20 billion parameter model got 17%. That's not very far behind 19%, which is the 120 billion parameter model."

Hallucinations and Model Accuracy

A significant concern raised is the issue of hallucinations—instances where AI models generate inaccurate or fabricated information.

• 120 Billion Parameter Model: Hallucinates in 49% of cases on the Person's QA benchmark.
• 20 Billion Parameter Model: Hallucinates in 53% of cases.

These figures are considerably higher compared to OpenAI’s proprietary models, with one version showing only a 16% hallucination rate. The host discusses potential reasons, including the reduced parameters in open-source models leading to less world knowledge and increased hallucinations.

[19:45] "OpenAI's new model does hallucinate much more than its latest, you know, 03 or 04 mini models. So that is not a particularly fantastic statistic."

OpenAI's Licensing and Data Practices

The podcast addresses OpenAI's decision to release these models under the Apache 2.0 license, one of the most permissive open-source licenses, allowing companies to monetize the models without restrictions.

[24:00] "They are releasing both of these models under the Apache 2.0 license. So this is really considered as one of the most, I guess like lenient licenses. It will allow companies to monetize this model, right?"

However, OpenAI distinguishes its "open models" from fully open-source models by not releasing the training data, likely due to legal concerns over copyrighted material.

[27:30] "Unlike fully open source models, what's the difference? They're calling an open model, but it's not totally an open source... They are not going to release the training data that they use to create their models."

Microsoft's Integration of Open Models into Windows

Shifting focus, the host highlights Microsoft's initiative to integrate OpenAI's smallest model into Windows 11 users via Windows AI Foundry. This integration aims to provide AI capabilities directly on consumer PCs, supporting tasks like code execution and tool usage.

[34:15] "Microsoft is basically bringing their smallest model... It's going to be for any Windows 11 users, which is pretty interesting."

Key features include:

• Tool Savvy and Lightweight: Optimized for tasks requiring autonomy.
• Efficient Performance: Runs on a range of Windows hardware, with support for more devices forthcoming.
• Versatility: Suitable for building autonomous assistants and embedding AI into various workflows, even in environments with limited bandwidth.

The host expresses excitement about the accessibility and potential applications of this integration, noting that it will be available via Hugging Face for developers and companies to utilize.

[38:50] "It's really a gift to the world and I'm sure OpenAI has more exciting things up their sleeve like GPT5 that'll probably blow this out of the water."

Conclusion and Future Outlook

Wrapping up, the host reflects on the current state and future prospects of open-source AI models. While OpenAI's latest open models represent a significant step forward, challenges like hallucinations remain areas for improvement. The collaboration with Microsoft signifies a broader trend of integrating AI more deeply into everyday technologies, promising increased accessibility and innovation.

[45:20] "What you actually need if you want to run this and this will be starting on Tuesday, but it'll be able to run on most consumer PCs and laptops... So really, really impressive."

The episode concludes with anticipation for forthcoming advancements, including potential releases like GPT-5, which may further elevate the capabilities and applications of AI models.

Notable Quotes

• Host [00:00]: "This is actually really big news because this is the first time in five years that they've actually dropped any open source models back to GPT2."

• Host [04:30]: "These are, these things aren't very far apart. It definitely did better than the O3 Midi model, which only got 2,000."

• Host [08:50]: "Tools basically mean they gave the AI model things like calculators and apps and like different tools. So like it is completing the tasks. Yes, but it's able to rely on like actual hard software to get good results."

• Host [12:15]: "I was actually really impressed that the 20 billion parameter model got 17%. That's not very far behind 19%, which is the 120 billion parameter model."

• Host [19:45]: "OpenAI's new model does hallucinate much more than its latest, you know, 03 or 04 mini models. So that is not a particularly fantastic statistic."

• Host [24:00]: "They are releasing both of these models under the Apache 2.0 license. So this is really considered as one of the most, I guess like lenient licenses. It will allow companies to monetize this model, right?"

• Host [27:30]: "Unlike fully open source models, what's the difference? They're calling an open model, but it's not totally an open source... They are not going to release the training data that they use to create their models."

• Host [34:15]: "Microsoft is basically bringing their smallest model... It's going to be for any Windows 11 users, which is pretty interesting."

• Host [38:50]: "It's really a gift to the world and I'm sure OpenAI has more exciting things up their sleeve like GPT5 that'll probably blow this out of the water."

• Host [45:20]: "What you actually need if you want to run this and this will be starting on Tuesday, but it'll be able to run on most consumer PCs and laptops... So really, really impressive."

This comprehensive summary encapsulates the key discussions from the episode, providing insights into OpenAI's strategic release of open-source models, their performance metrics, challenges like hallucinations, licensing implications, and Microsoft's integration efforts. Listeners and enthusiasts alike will find this overview valuable for understanding the current dynamics and future directions in the realm of artificial intelligence.