Cybermetrics and Groundhog Day

Summary

More or Less – "Cybermetrics and Groundhog Day"

Podcast: More or Less
Host: Tim Harford (BBC Radio 4)
Episode Date: February 17, 2012
Theme: A critical look at the use—and misuse—of statistics, including the validity of Google search result numbers and the accuracy of a famous weather-predicting groundhog.

---

Episode Overview

This episode examines two topics where numbers are at the heart of widespread beliefs and reporting:

• The reliability of using Google search result counts as indicators of popularity or extent.
• The statistical analysis of Groundhog Day predictions and whether Punxsutawney Phil is more accurate than chance.

With the show's signature wit and skepticism, Tim Harford and producer Ruth Alexander dissect common misconceptions and explain where the numbers can mislead.

---

Key Discussion Points & Insights

1. Can Google Search Results Be Trusted?

[01:01–05:51]

• Listener question: Is quoting the number of Google search results a valid indicator of popularity?
• Illustrative Example: Ruth “Googles” Tim Harford’s name on different devices and accounts, yielding widely varying results: 285,000, 350,000, and even 835,000 hits on the same day.
  • Ruth Alexander [02:03]: “The number of results you'll get depends on the computer you're using and which copy of Google you're using.”
• Key Takeaway: There are multiple copies of Google's search infrastructure worldwide. Search queries are routed to whichever is least busy, leading to inconsistent results.
• Name Effect: Popular names or ambiguous terms further inflate hit counts, as shown in Ruth's case with "Ruth Alexander," a combination of two common first names.
  • Ruth Alexander [02:36]: "I'd love to say that I am many times more famous than you, Tim, but... my name is made up of two popular first names. And that's my secret."
• Web Coverage:
  • Only about 17% of the web was indexed by major search engines according to a study from 11 years prior [03:25]. This is due to search engines' methods of crawling (“spidering”) from known sites by following links.
  • A newly created, unlinked website is likely invisible to Google.
  • Tim Harford [03:53]: "That number's 11 years old though, which in web terms is 11 ice ages ago."
  • Ruth Alexander [03:59]: "The web is a fast moving target, growing rapidly... they don't know [if coverage is better]."
• Hit Count Accuracy:
  • For one-word queries, search result counts can be reasonable.
  • Adding more words (multi-word queries) drastically reduces accuracy.
    • Ahmet Uyaj [04:44]: "80% of the times the estimates were reliable... with only 10% errors... for two-word queries, the accuracy is reduced almost by half."
  • Even when Google claims, say, 29 results, only a subset (like 21) may actually be viewable.
  • Ruth Alexander [05:30]: "Experts reckon they cut some corners in the name of speed. And the hit numbers they come up with are estimates, not hard numbers."
• Conclusion:
  • Tim Harford [05:51]: "Sounds like we need to take these figures with a pinch of salt... Journalists might be revealing more about their journalism than about the subject in question."

---

2. Groundhog Day: Can Phil the Groundhog Predict the Weather?

[05:51–09:02]

• Background: On February 2nd, “Punxsutawney Phil” emerges; his seeing (or not seeing) his shadow supposedly predicts the next six weeks’ weather.
• Comedic Take:
  • Tim Harford [07:01]: “This is the one time when radio really fails to capture the true excitement of a large squirrel predicting the weather.”
  • Ruth Alexander [06:29]: “There are people who understand his snuffles.”
• Statistical Analysis:
  • Paul Summers, Professor of Economics at Middlebury College, analyzed weather data from 1950 to 1999—comparing minimum/maximum temperatures and snowfall in the six weeks before and after Groundhog Day.
    • Method: Check for breaks in average temperature trends after February 2 to judge “early spring.”
    • Results:
      • Paul Summers [08:31]: “Phil was correct more than 70% of the time. He was correct 87% of the time when he did see his shadow.”
      • This challenges the common belief that Phil is only accurate as a coin toss (50/50).
      • Paul Summers [08:46]: “He was significantly correct more than 50% of the time.”
  • Caveat: The show does not dig into methodological issues (such as definition of “correct” or local microclimates) but presents the result as surprising and (lightly) credence-enhancing.
• Final Words:
  • Tim Harford [09:02]: “No sign of spring in Pennsylvania for some time, then. Hats off to Phil and Paul and thanks to the World Wide Web's ubiquitous Ruth Alexander.”

---

Notable Quotes & Memorable Moments

• Tim Harford [01:13]: “A quick online search of essay writing services returns more than 31 million hits. Clearly, these businesses are thriving.”
• Ruth Alexander [02:03]: “The first lesson of search engine accuracy is that the number of results you'll get depends on the computer you're using.”
• Listener via Ruth Alexander [03:25]: “About 11 years ago, some researchers tried to assess exactly how much of the web was to be found in the indexes of major search engines. And they came up with the answer of about 17% of the web.”
• Ahmet Uyaj [04:44]: “When we try two word queries, then the accuracy is reduced almost by half.”
• Tim Harford [05:51]: “Sounds like we need to take these figures with a pinch of salt.”
• Tim Harford [07:01]: “This is the one time when radio really fails to capture the true excitement of a large squirrel predicting the weather.”
• Paul Summers [08:31]: “Phil was correct more than 70% of the time. He was correct 87% of the time when he did see his shadow.”
• Ruth Alexander [08:42]: “That's an impressive set of results.”

---

Key Timestamps

• 00:13: Introduction to episode and podcast’s purpose
• 01:01: Introduction to search result accuracy segment
• 03:25: Explanation of web indexing (only 17% indexed by search engines)
• 04:44: Ahmet Uyaj on error rates in hit counts
• 05:51: Transition to Groundhog Day investigation
• 07:43: Paul Summers explains Groundhog Day analysis methodology
• 08:31: Groundhog prediction accuracy rates revealed

---

Final Thoughts

With humor and clear-eyed skepticism, this episode of More or Less uncovers the profound limitations behind two widely quoted statistics: Google search result counts and the predictive powers of a famous rodent. While the former turns out to be highly unreliable, the latter—surprisingly—may be better than flipping a coin. The show serves as a reminder to question the easy numbers and look for the real story behind the stats.

Transcript

[BBC Announcer]

Thank you for downloading from the BBC. For details of our complete range of podcasts and our terms of use, go to bbcworldserveys.com podcasts.

[Tim Harford]

You're listening to a download from the BBC. This is more or less the mathematical icing on the cake of life. We make an edition of the program for BBC Radio 4, but this is a download of a special edition broadcast on the World Service. Hello, and welcome to More or Less on the BBC World Service. We're your weekly guide to the numbers in the news and in life. And I'm Tim Harford. This week, it's Groundhog Day again.

[Groundhog Day Announcer]

As I look at the crowd on Gobbler's Knob, many shadows do I see. Six more weeks of winter it must be.

[Tim Harford]

We put the famed weather predicting prowess of a North American movie star, groundhog, to the statistical test. But first, a listener writes.

[Listener]

I'm still seeing articles quoting Google search numbers as some kind of indicator for the extent or popularity of a topic. Is there any justification for using these numbers?

[Tim Harford]

This is the kind of thing they're talking about.

[Ruth Alexander]

A quick online search of essay writing services returns more than 31 million hits. Clearly, these businesses are thriving.

[Tim Harford]

So can you be sure of the popularity of someone or something? But based on the number of results that are thrown up when you type that name or topic into an Internet search engine, Ruth Alexander's been looking at this. Hello, Ruth.

[Ruth Alexander]

Hello.

[Tim Harford]

The answer is.

[Ruth Alexander]

Do you Google yourself a lot, Tim?

[Tim Harford]

No. No, not that often. Trying to give it up?

[Ruth Alexander]

Well, I just googled you and got 285,000 hits.

[Tim Harford]

Well, I Googled myself half an hour ago. Well, yeah, okay, I got. Got 350,000 hits.

[Ruth Alexander]

Well, interestingly, when the editor puts your name in, he gets 835,000 hits.

[Tim Harford]

Ruth, you've broken Google.

[Ruth Alexander]

No, I haven't. The first lesson of search engine accuracy is that the number of results you'll get depends on the computer you're using and which copy of Google you're using.

[Tim Harford]

I thought there was only one copy of Google.

[Ruth Alexander]

So did I. But apparently there are several copies of Google in the world. And your search query will be dispatched to whichever version is the least busy now. It's also worth mentioning early on, I think that I Googled myself and got 68 million hits.

[Tim Harford]

68 million. That's a lot more than me.

[Ruth Alexander]

Well, I'd love to say that I am many times more famous than you, Tim, but the sad truth is I think that my name, Ruth Alexander, is made up of two popular first names. And that's my secret.

[Tim Harford]

There we go. Well, at least we know the search engines are, I guess, counting the web pages correctly, even if the web pages themselves may be misleading.

[Ruth Alexander]

Actually, we don't know that. Search engine companies like Google and Yahoo, for example, have huge banks of computers called server farms which try to download all the information on the web so that they can then index it and refer to that index when you or I make a search. But here's a stat about that which amazed me. I was told about it by Professor Mike Thelwall of Wolverhampton University in the uk who counts things on the web for a job. Here's the amazing stat.

[Listener]

About 11 years ago, some researchers tried to assess exactly how much of the web was to be found in the indexes of major search engines. And they came up with the answer of about 17% of the web. And the reason is because search engines start with a few big websites and then they try and find new websites, mainly by following the links on existing websites. If you've got a fairly new website that no one links to yet, the chances are that Google won't know about you.

[Tim Harford]

And okay, that number's 11 years old though, which in web terms is 11 ice ages ago.

[Ruth Alexander]

Well, true, you might assume that the search engine companies must be getting better. But on the other hand, the web is a fast moving target, growing very rapidly. So whether more of the web is indexed now, they don't know.

[Tim Harford]

Any other surprises for us?

[Ruth Alexander]

There is one, actually. You might think that if Google tells you it's returned a million hits, there's a list of a million pages you could in principle go and view.

[Tim Harford]

You might indeed.

[Ruth Alexander]

Yes, but not so. A study that compared the hit counts of three search engines shows this quite clearly. The hit counts are pretty accurate when you search for one word. But each time you add a word to your search, the numbers get less and less accurate. Here's Ahmet Uyaj from Maliksha University in Turkey, who did the research.

[Ahmet Uyaj]

80% of the times the estimates were reliable, so they had only 10% errors in their estimation. But when we try two word queries, then the accuracy is reduced almost by half.

[Tim Harford]

Did they really count all the pages in each search?

[Ruth Alexander]

Yes, they did, because they were only submitting queries that came up with fewer than 1,000 results. In fact, I did it for myself. I searched for the first part of a limerick. There was a young man from Darjeeling who got on a bus. First of all, when you put that In, Google shows 15 results, but it invites you to click to see omitted results, then it says there are 29 in total, but in actual fact it can only show you 21 results.

[Tim Harford]

I mean, why? Surely Google can count to 29?

[Ruth Alexander]

Well, I think the supercomputers behind the scenes have to work very quickly, don't they? Mixing and matching lots of documents, throwing out spam and pages where the words surrounding my search term are the same. And in doing all this in less than a second or two, experts reckon they cut some corners in the name of speed. And the hit numbers they come up with are estimates, not hard numbers.

[Tim Harford]

Sounds like we need to take these pliggers with a pinch of salt. And I would recommend to any journalists before they quote them, they might be revealing more about their journalism than they are about the subject in question. Now to the meteorological forecasting powers of a creature made famous in the Hollywood film Groundhog Day. On Groundhog Day, Punxsutawney Phil, a groundhog comes out of hibernation to give his verdict on the weather for the next six weeks. If he sees his shadow, it's six more weeks of winter. If he doesn't, spring will come early. How anyone knows whether he's seen his shadow or not, I do not know.

[Ruth Alexander]

Oh, I do. There are people who understand his snuffles.

[Tim Harford]

Are there? Well, here's this year's prediction as translated by the people Hear ye, hear ye,

[Groundhog Day Announcer]

hear ye on Gobbler's Knob on this magnificent Groundhog Day, February 2, 2012. Phil Many shadows do I see six more weeks of winter. It must be.

[Tim Harford]

This is the one time when radio really fails to capture the true excitement of a large squirrel predicting the weather. But the more or less question surely is given the difficulties humans have forecasting the weather, how accurate can a groundhog possibly be?

[Ruth Alexander]

Well, Tim, you might be surprised. In the film Groundhog Day, weather presenter Phil Connors is skeptical like you, but in real life, a professor of economics has done some very interesting statistical analysis of Phil the Groundhog's meteorological assessments. Paul Summers of Middlebury College in the United States loves looking for patterns in data. He looked at the local weather records from 1950 to 1999.

[Paul Summers]

I collected data daily information on temperature six weeks before February 2, or Groundhog Day and six weeks thereafter. So I had a total of 84 observations on minimum temperature, maximum temperature, and also snowfall. And what I was curious to do with my three student co authors was to see whether or not there was a break in the Trend line after February 2nd. For example, if the line suddenly shifts up, in other words, indicating that there's an increase in temperature on average. That would be an indication of an early spring.

[Ruth Alexander]

The next step was to look back to see in which years Phil had seen his shadow and in which years he didn't.

[Paul Summers]

Bottom line is Phil was correct more than 70% of the time. He was correct 87% of the time when he did see his shadow.

[Ruth Alexander]

That's an impressive set of results.

[Paul Summers]

We thought so, because I think National Geographic, other detractors of Punxsutawney Phil believe that he's no better than what you would get if you were flipping a coin. But indeed, he was significantly correct more than 50% of the time.

[Tim Harford]

No sign of spring in Pennsylvania for some time, then. Hats off to Phil and Paul and thanks to the World Wide Web's ubiquitous Ruth Alexander. If there's a statistical claim you want us to investigate, email more or lessbc.co.uk and you can download our podcast via our website, bbcworldservice.com more or less until next week. Goodbye. More or Less was presented by me, Tim Harford, the Financial Times undercover economist. The producer was Ruth Alexander and the editor, Richard Varden.

[BBC Announcer]

There are dozens of different podcasts now available from the BBC, including news, documentaries, science, business, arts and sports. For details of them all, go to bbcworldservice. Com Podcasts.