The dawn of search engines

Summary

Summary of "The Indicator from Planet Money" Episode: The Dawn of Search Engines

Release Date: May 26, 2025
Host: Adrian Ma (NPR)

---

Introduction

In this special guest episode of The Indicator from Planet Money, host Adrian Ma presents an excerpt from the Canadian Broadcasting Corporation's series "Who Broke the Internet?" Hosted by Cory Doctorow, the series delves into criticisms of the modern Internet and explores potential solutions. This particular episode focuses on the inception and evolution of search engines, highlighting how they transformed the way we navigate the vast expanse of the Internet.

Early Search Engines and Their Limitations

Steven Levy begins by painting a picture of the early Internet landscape, where users struggled to locate relevant information due to primitive search engine technologies. Early search engines primarily operated by scanning for pages containing the exact words inputted by users, prioritizing pages with higher word counts. However, this approach was often ineffective:

"If you wanted your page to rate high on the search results for a query like Mexican food, you could keyword stuff it by adding the words Mexican food a thousand times in tiny white on white type to the bottom of the page."
— Steven Levy [01:57]

This tactic exploited the simplistic algorithms of early search engines, leading to poor-quality search results. Users often had to employ complex and arcane search syntax (e.g., using minus signs to exclude unwanted terms) to filter out spam and irrelevant content, making the search process cumbersome and inefficient.

The Emergence of Google and PageRank

The breakthrough came with the introduction of Google, which revolutionized search engine technology through the development of the PageRank algorithm. Clive Thompson recounts his first encounter with Google:

"We were using really awful search engines that we thought were pretty good. And you saw Google and all of a sudden the film cleared from our eyes. We thought, wow, you could actually find what you were looking for on the Internet right away. Bang."
— Clive Thompson [03:12]

Steven Levy, author of How Google Thinks, Works and Shapes Our Lives, elaborates on his journalist's pursuit of Google's founders:

"So he tracked down their PR person."
— Steven Levy [03:27]

Thompson describes meeting Google's founders, Larry Page and Sergey Brin, and witnessing their innovative approach firsthand:

"Larry was dressed like a Viking and Sergey was dressed like a cow. He had these big plastic udders coming out from his chest."
— Clive Thompson [04:07]

The PageRank algorithm marked a significant departure from previous search methodologies. Instead of merely counting keyword occurrences, PageRank analyzed the links between websites to determine their importance and relevance. This method provided more accurate and meaningful search results.

"Page rank is the algorithm behind the magic. A new way that Google had developed to deliver its search results. It was totally audacious..."
— Steven Levy [04:28]

Thompson highlights the audacity of Google's approach:

"Audacious because the way they were able to locate the best result for your query was by basically not basically literally downloading the whole web, all of it, which some people thought was ridiculous. But Larry Page understood this could be done."
— Clive Thompson [04:39]

With access to powerful servers at Stanford University, Page and Brin built Google's first server setup using Lego bricks—symbolizing both cost-effectiveness and creativity:

"They set up their first server, building the case for it out of Lego. Yeah, Lego, because it was cheap and cool and and colorful like the eventual Google logo."
— Steven Levy [05:13]

Google’s Philosophy and Growth

Google's foundational philosophy emphasized simplicity and user-centric design. Clive Thompson notes the company's guiding principle:

"There's the very famous motto that Google had, was 'Don't be evil,' which was coined by an engineer..."
— Clive Thompson [06:59]

This motto encapsulated Google's commitment to providing high-quality search results without succumbing to unethical practices, such as compromising search quality for advertising revenue. Steven Levy reflects on Google's influence:

"Google's world was our world. You know, things didn't exist, but it wasn't on the web."
— Clive Thompson [07:27]

As Google gained popularity for its effective search capabilities, its success was intrinsically linked to the overall usability of the Internet:

"It's like when Google did well, we all did well because we could find the stuff we were looking for, right?"
— Steven Levy [07:33]

The Introduction of Ads and AdWords

The introduction of advertising marked a pivotal shift in Google's business model. Initially, Google founders were critical of traditional advertising methods:

"When Google met advertising, everything changed."
— Steven Levy [07:52]

Clive Thompson elaborates on the founders' aversion to advertising:

"Larry and Sergey actually hated advertising. And the paper they wrote, talking about PageRank, they had a little thing in the end saying, 'Boy, we hate advertising. Terrible.'"
— Clive Thompson [07:55]

Despite their reservations, the need to generate revenue led to the development of a novel advertising platform—AdWords:

"So then they figured, well, maybe there's a way to do ads that would be terrible."
— Steven Levy [08:30]

AdWords introduced a system where ads were seamlessly integrated into search results through an auction-based model:

"It was called AdWords. So what it would do was when you put a query into the search engine, it would start an auction instantly..."
— Clive Thompson [08:57]

This innovation allowed advertisers to bid on search terms relevant to their products or services, ensuring that ads were both targeted and unobtrusive:

"If you searched for Mexican food on Google back in the early aughts... with AdWords, ads would appear in a colored bar atop the normal page ranked results."
— Steven Levy [09:07]

Clive Thompson underscores the benefits of AdWords:

"You get the same results but you get a little extra little, you know, set of possible results that actually might even help you if you were looking to buy something related to the search term and the advertisers would get an audience which you know was primed to buy something."
— Clive Thompson [09:44]

AdWords proved to be a win-win scenario for Google, advertisers, and users by maintaining search result quality while generating revenue.

Conclusion

The episode concludes by acknowledging the transformative impact of Google's search engine and its advertising model on the Internet ecosystem. While AdWords initially enhanced the search experience, Steven Levy hints at future challenges:

"That is, until it didn't."
— Steven Levy [10:11]

This statement foreshadows the complexities and potential pitfalls associated with search engine advertising, setting the stage for further exploration in subsequent discussions.

---

Credits:
Original production by Matt Muse, A.C. Rowe, and Cory Doctorow. Coordinating Producer: Roshni Nair. Mixing and Sound Design: Julian Uzieli. Story Editing: Veronica Simmons. Executive Producer: Nick McCabe. Indicator version produced by Cooper Katz McKim and engineered by Sina Alafredo. Edited by Kate Cannon.

---

For more insights from the CBC's "Who Broke the Internet?" series, visit the show notes or your preferred podcast platform.

Transcript

[NPR Host]

Npr.

[Adrian Ma]

This is the indicator from Planet Money. I'm Adrian Ma. Today we've got a special guest episode for you. It's an excerpt from a series called who Broke the Internet? From our colleagues up north at the Canadian Broadcasting Corporation. It's from their Understood feed at CBC Podcasts. And the host is Cory Doctorow. You might have heard of him. He's a journalist, blogger, science fiction writer, and Internet commentator, and his series digs into his criticisms on the state of the modern Internet and what we can do about it. So in this excerpt you're going to hear after the break, Cory's going to talk about how Search engines got started.

[NPR Host]

This message comes from Charles Schwab. When it comes to managing your wealth, Schwab gives you more choices like full service, wealth management, and advice when you need it. You can also invest on your own and trade on thinkorswim. Visit schwab.com to learn more.

[Apple Ad]

This message comes from Apple Card Somewhere in the world, an Apple Card user is getting 3% daily cash back on the purchase of an iPhone 16 at Apple. That's not all. They also earned 2% back on the new shoes they bought using Apple Pay. Visit Apple Co CardCalculator and see how much daily cash back you can earn. Subject to credit approval, Apple Card issued by Goldman Sachs Bank USA, Salt Lake City branch terms and [e-mail] this message.

[NPR Host]

Comes from Amazon Business With Smart Business Buying, get everything you need to grow in one familiar place, from office supplies to IT essentials and maintenance tools ready to bring your visions to life. Learn [e-mail].

[Steven Levy]

Eventually, the Internet got too big for people to know where everything was, so waves of clever people created search engines. Early search engines just looked for pages containing the words you typed, giving priority to pages that contained more of those words. This worked okay, but when it failed, boy, did it ever fail badly. If you wanted your page to rate high on the search results for a query like Mexican food, you could keyword stuff it by adding the words Mexican food a thousand times in tiny white on white type to the bottom of the page. The primitive search engines would count these all up and conclude that your page was the most important Mexican food resource in the world. Which is useless because most hungry people aren't looking for a site that just has the words Mexican food a thousand times. To weed these bad hits out of your search results, you'd have to master all kinds of arcane search engine syntax so you could exclude words and phrases, putting minus signs in front of the kind of junk that was typical of spam sites. It took forever, but those search engines were all we had, so we kept using them.

[Clive Thompson]

We were using really awful search engines that we thought were pretty good. And you saw Google and all of a sudden the film cleared from our eyes. We thought, wow, you could actually find what you were looking for on the Internet right away. Bang.

[Steven Levy]

This is Steven Levy. He wrote the canonical history of Google in the How Google Thinks, Works and Shapes Our Lives. Back in the early 90s, Stephen was Newsweek's tech reporter, and he knew right away that Google was a story.

[Clive Thompson]

When I first saw the Google search engine, my mind was blown.

[Steven Levy]

So he tracked down their PR person.

[Clive Thompson]

I called her up, I said, I gotta meet these guys.

[Steven Levy]

These guys being Google's founders, two Stanford grad students named Larry Page and Sergey Brin. So Stephen booked a ticket from New York to San Francisco and flew out to meet them.

[Clive Thompson]

It was near Halloween. Everyone was dressed in Halloween costume. So Larry was dressed like a Viking and Sergey was dressed like a cow. He had these big plastic udders coming out from his chest. So the Viking and the cow took me into a little room and explained how PageRank works.

[Steven Levy]

Page rank is the algorithm behind the magic. A new way that Google had developed to deliver its search results. It was totally audacious, and I say.

[Clive Thompson]

Audacious because the way they were able to locate the best result for your query was by basically not basically literally downloading the whole web, all of it, which some people thought was ridiculous. But Larry Page understood this could be done.

[Steven Levy]

All Larry and Sergey needed to pull it off was big, powerful servers, which.

[Clive Thompson]

They could get because they had access to servers at Stanford. And they would literally hijack them as they came in the loading dock, meant for other departments.

[Steven Levy]

With all this yoinked power, they set up their first server, building the case for it out of Lego. Yeah, Lego, because it was cheap and cool and and colorful like the eventual Google logo.

[Clive Thompson]

And then they would do math. This was like Sergey's specialty to analyze all the links that happen between websites and figuring out what important websites link to other sites. So that way you could filter out which were the perfect answers for the queries you put into the search field. And that turned to be just a quantum leap, better than the previous things.

[Steven Levy]

The early Internet wasn't perfect. Google came into existence because the search landscape sucked. But the point is, Google could come into existence. There was oxygen and sunshine that reached the forest floor, so new things could grow. If you could code, if you had some money, a computer or two. A couple of bins of Lego, you could. You were allowed to build something new that made the web better. Clive Thompson Again, they start off by having a really good search engine, the best search engine. And the reason why it's so good is that they are in ferocious competition with, like, 12 different search engines. And they have to be better than everyone else. They have to fight to the top, and they do that. Just be the best, make the best product, and people will switch. And we did. We all switched to Google because it worked, and that mattered to Google in their early days.

[Clive Thompson]

And there's the very famous motto that Google had, was don't be evil, which was coined by an engineer when someone from HR had a meeting. And, you know, so what is our motto? What is our. What is our. Our. Our value? And, you know, one guy said, what are you talking about? Just like, don't be evil. You know, I mean, that's it.

[Steven Levy]

So back in 1999, Stephen wrote that first story for Newsweek, and he kept writing about Google.

[Clive Thompson]

Google's world was our world. You, you know, things didn't exist, but it wasn't on the web.

[Steven Levy]

It's like when Google did well, we all did well because we could find the stuff we were looking for, right?

[Clive Thompson]

And that came into play even more when they came up with their first big successful business product.

[Steven Levy]

That was ads.

[Clive Thompson]

Yeah, yeah.

[Steven Levy]

When Google met advertising, everything changed. The introduction of ads to Google marked a major shift. Emmanuel Goldstein Again, Google, in their early days, you didn't have ads on the front page. It was relatively fast, and they just seemed like cool people inventing cool things.

[Clive Thompson]

Larry and Sergey actually hated advertising. And the paper they wrote, talking about PageRank, they had a little thing in the end saying, boy, we hate advertising. Terrible.

[Steven Levy]

Stephen's talking about an article the Google founders published in a journal in 1998. And in it, they explicitly wrote that quote, advertising income often provides an incentive to provide poor quality search results.

[Clive Thompson]

But they had to make money. So then they figured, well, maybe there's a way to do ads that would be terrible.

[Steven Levy]

So in 2000, the founders assigned an engineer to figure this out, one of Google's first eight employees, and he came up with something.

[Clive Thompson]

It was called AdWords. So what it would do was when you put a query into the search engine, it would start an auction instantly, and, you know, your search term would be auctioned off to advertisers who wanted to sell a product that was, you know, related.

[Steven Levy]

If you searched for Mexican food on Google back in the early aughts, the results would have been the most linked to websites for Mexican restaurants in your area and reviews from local blogs and newspapers. And now with AdWords, ads would appear in a colored bar atop the normal page ranked results.

[Clive Thompson]

Yeah, you get the same results but you get a little extra little, you know, set of possible results that actually might even help you if you were looking to buy something related to the search term and the advertisers would get an audience which you know was primed to buy something. And it worked for Google, it worked for advertisers, and most of all, it worked for users.

[Steven Levy]

That is, until it didn't.

[Adrian Ma]

Thanks to our colleagues over at the Canadian Broadcasting Corporation. If you want to hear more of who Broke the Internet, you can find a link in our show notes or find it wherever you listen to podcasts. The original version of this episode was written and produced by Matt Muse, A.C. rowe and Cory Doctorow. Roshni Nair is the coordinating producer. Julian Uzieli did the mixing and sound design. Story editing by Veronica Simmons. The executive producer is Nick McCabe locos the indicator version of this episode was produced by Cooper Katz McKim and engineered by Sina Alafredo. Kate Cannon is our editor and the indicators of production of npr.

[NPR Host]

This message comes from Mint Mobile if you're tired of spending hundreds on big wireless bills, bogus fees and free perks, Mint Mobile might be right for you with plans starting from 15 bucks a month. Shop plans [e-mail] Switch upfront payment of $45 for 3 month 5 gigabyte plan required. New customer offer for first 3 months only then then full price plan options available, taxes and fees extra. See Mint Mobile for details. This message comes from Warby Parker. What makes a great pair of glasses at Warby Parker? It's all the invisible extras without the extra cost, like free adjustments for life. Find your [e-mail] or visit one of their hundreds of stores around the country. This message comes from Mint Mobile. Mint Mobile took what's wrong with wireless and made it right. They offer premium wireless plans for less and all plans include high speed data, unlimited talk and text and nationwide coverage. See for [e-mail] Switch.