WS MoreOrLess: Testing the PISA test

Summary

More or Less: Testing the PISA test

Podcast Episode Summary

Podcast: More or Less<br> Host: Ruth Alexander (BBC Radio 4, World Service)<br> Date: December 7, 2013<br> Theme: Scrutinizing the Numbers Behind International Education League Tables

---

Episode Overview

This episode examines the PISA test, an influential international assessment of 15-year-old students’ educational abilities. The show dives into what the results mean, how the numbers are produced, their limitations, and the broader implications for education policy. Together with statisticians and quiz questions, the episode critically analyzes whether PISA’s rankings should be trusted and how they are used by the media and policymakers.

---

Key Discussion Points and Insights

1. What is the PISA Test?

• Introduction (00:12 – 00:34):
  Host Ruth Alexander introduces the international league table where the UK appears to be falling behind global rivals in tests taken by 15-year-olds.
• Scope (00:39 – 02:21):
  • PISA is the OECD’s Programme for International Student Assessment.
  • Every three years, 15-year-old students in 65 countries sit standardized tests in maths, reading, and science.
  • Results fuel global education rankings, influencing media and policymakers.
  • Top performers: Shanghai, Singapore, Hong Kong, Taiwan, South Korea.
  • Bottom performers: Peru, Indonesia, Brazil, Tunisia (the only African participant).

2. Sample Test Questions and Performance

• Question 1 (Level 2) — Average Speed (01:12 – 02:11):
  "Helen rode 4km in 10 minutes and 2km in 5 minutes. Did she ride faster, slower, or at the same speed on each part?"

  • Correct answer: Same speed.
  • Performance: Shanghai, Hong Kong, Singapore: 96%; overall average: 77%.

• Question 2 (Level 3) — Engine Size Comparison (03:12 – 03:45):
  "Which engine is smaller: 1.79 litres or 1.783 litres?"

  • Correct answer: 1.783 litres is smaller (rounds to 1.78).
  • Performance: Shanghai: 89%; overall OECD average: 55%.

• Question 3 (Level 6) — Calculating Average Speed for Different Routes (06:26 – 07:25):
  "Helen cycles 4km in 9 minutes, returns 3km in 6 minutes. What's her average speed in km/h?"

  • Correct answer: 28 km/h.
  • Performance: Shanghai: ~30%; UK, France, Austria: only 3%.
  • Notable Moment: Ruth works through the math live, demonstrating the question's challenge.

3. Criticism and Limitations of the PISA Test

• Statistical Methods and Plausible Values (02:57 – 04:51):

  • David Spiegelhalter (University of Cambridge) raises concerns:

    "I think that the whole PISA enterprise could do with a lot more external review in terms of its statistical methods, which are unbelievably complex and rather opaque." (02:57)

  • PISA does not give all students all questions; instead, each student takes a subset, and models extrapolate their scores on unattempted questions using “plausible values.”
  • Spiegelhalter warns:

    "These plausible values, I think, are open to some criticism because they're predicted conditional on knowing the difficulty of the questions... assuming that the difficulty is the same for all students around the whole world." (04:35)

• Assumptions About Question Equivalence (04:51 – 06:04):

  • There’s debate about whether tests are truly comparable between countries.
  • Andreas Schleicher (OECD) responds:

    "The model is always an approximation of reality... The question is, does the model fit the reality such that there is no distortion of the results?" (05:26)

4. The Meaning (and Danger) of Rankings

• Focusing Too Much on Rankings (06:04 – 08:10):

  • Even Schleicher cautions about overemphasizing precise country positions due to margin of error.
  • The broader patterns—some countries consistently scoring higher—are meaningful.
  • Beware of “PISA shock”: when countries like Germany, Denmark, and Norway made hasty policy changes after surprisingly low rankings in 2001.

• The Limits of Comparisons (08:10 – 08:47):

  • Spiegelhalter thinks the math questions are useful but warns:

    "Aligning policy along a single performance indicator or even one or two performance indicators can cause all sorts of distortions in the system... What it should do is create a wider debate about what do you want out of an education system." (08:10)

  • He notes that high-scoring countries like Korea fare poorly on student happiness:

    "Korea...does very well; the kids are miserable—65 out of 65 in the league table for happiness in schools." (08:41)

5. Conclusions

• International comparisons are valuable for broad insights but should not drive all education policy.
• Test results should open debate, not dictate it—a view shared by both critics and proponents.
• Numbers must be interpreted with caution.

---

Notable Quotes and Memorable Moments

• David Spiegelhalter on Complexity:

  "I think that the whole PISA enterprise could do with a lot more external review in terms of its statistical methods, which are unbelievably complex and rather opaque." (02:57)

• Charlotte MacDonald on Achievement:

  "96% of 15 year olds in Shanghai, Hong Kong and Singapore manage this level of question. In the test, the average was 77%." (02:11)

• Andreas Schleicher on Modeling Reality:

  "The model is always an approximation of reality... does the model fit the reality such that there is no distortion of the results?" (05:26)

• David Spiegelhalter on Policy and Wellbeing:

  "Aligning policy along a single performance indicator or even one or two performance indicators can cause all sorts of distortions in the system." (08:10)
  "Korea... the kids are miserable—65 out of 65 in the league table for happiness in schools." (08:41)

---

Key Timestamps

• 00:39 — PISA test overview and rankings
• 01:12 — Sample test question (Level 2)
• 02:11 — International performance discussion
• 02:57 — Statistical criticism: Interview with David Spiegelhalter
• 03:12 — Sample test question (Level 3)
• 04:35 — Critique on “plausible values” and test construction
• 05:26 — Andreas Schleicher defends the methodology
• 06:26 — Sample test question (Level 6)
• 07:39 — Limits of country rankings; “PISA shock”
• 08:10 — David Spiegelhalter on policy risks and student wellbeing
• 08:47 — Synthesis and recommendations

---

Tone and Style

The episode maintains the More or Less ethos: curious, skeptical, light, and numerate, with a playful approach to math quizzes mixed with serious debate on educational policy and statistics.

---

Summary for the Uninitiated

If you haven’t listened:
This episode of More or Less uses the PISA global education rankings as a springboard to discuss the reliability and meaning of large-scale comparative data in education. With live math questions, interviews, and thoughtful critique, the program demonstrates why numbers in headlines don’t always tell the full story, and why caution and debate are essential ingredients when interpreting international league tables.

Transcript

[BBC Announcer]

Thank you for downloading from the BBC. For details of our complete range of podcasts and our terms of use, go to bbcworldserveys.com podcasts

[Ruth Alexander]

this is the short edition of More or Less, first broadcast on the BBC World Service. Hello and welcome to More or Less on the BBC World Service. I'm Ruth Alexander. This week, an international education league table creates waves. The UK is falling behind global rivals in international tests taken by 15 year olds.

[Andreas Schleicher]

U.S. test scores remain stagnant, while other countries see rapid rise.

[Ruth Alexander]

15 year old school pupils in 65 countries sit tests and every three years the results are compared. This time, maths was the main focus, but there are some who say that the people who design the test must try harder. We'll be exploring these concerns, but before we go any further, it's if you're anything like the More or Less team, you'll want to prove to yourself that you can answer a maths question aimed at a 15 year old. So, introducing quizmaster Charlotte MacDonald.

[Charlotte MacDonald]

The PISA questions are divided into six levels of difficulty. So I thought if you're listening to More or Less, you're not daft. So we'll plunge straight in. At level two, Helen's got a new bike. It has a speedometer which tells her the distance she travels and her average speed. Speed for the journey, Helen rode four kilometres in the first 10 minutes and then two kilometres in the next five minutes. Which one of the following statements is correct? Is it? A. She rode faster on average in the first part of the ride than in the second. B, she rode at the same speed the whole time on average. C, she rode more slowly in the first part of her journey than in the second part. D, you can't tell.

[Ruth Alexander]

Well, she rode 4 km in 10 minutes and then half that distance in half the time, 2 kilometres in 5 minutes. So the answer is B, she rode at the same speed the whole time, on average.

[Charlotte MacDonald]

Correct. Now, 96% of 15 year olds in Shanghai, Hong Kong and Singapore manage this level of question. In the test, the average was 77%.

[Ruth Alexander]

Now, the five top performers in this big international test are all in Asia. Shanghai is at the top, top of the class, followed by Singapore, Hong Kong, Taiwan and South Korea. Countries like the US and the UK are middling and Peru languishes at the bottom of the league with the likes of Indonesia, Brazil and Tunisia not far behind. Tunisia, by the way, is the only African country that at the moment takes part in these voluntary tests. The results from pisa, the Programme for International Assessment run by the oecd are taken as gospel by the media and many policy makers. But should they be?

[David Spiegelhalter]

I think that the whole PISA enterprise could do with a lot more external review in terms of its statistical methods, which are unbelievably complex and rather opaque.

[Ruth Alexander]

Professor David Spiegelhalter from the University of Cambridge's Statistical Laboratory.

[Charlotte MacDonald]

But before we go on, question number two. This is a level three question, so it's a little harder. Chris finds two cars at a local car dealer. They each have different size engines. Which has the smallest, the one which has a 1.79 litre capacity or the one which has a 1.783 litre capacity?

[Ruth Alexander]

Well, this one's quite easy. The answer is 1.783. If we rounded it to two decimal places, it would round down to 1.78.

[Charlotte MacDonald]

Correct. Now, Shanghai were the top scorers at this level. 89% of 15 year olds got this level of question. Right now, the OECD average was 55%.

[Ruth Alexander]

Thanks, Charlotte. Around 4,000 children in each country take part in the test, half a million in all. They get about two minutes per question. The test is made up of about 400 questions and each child is given about 50 of them to answer. It means that the children answer different selections of questions. Peter designs the test like this, to make sure a breadth of subject areas and skills are covered. It makes sense, but it also creates a problem. It PISA has to use a statistical model to extrapolate from each child's answers how they would have answered the other 350 questions had they been given them. David Spiegelhalter says this practice raises its own questions.

[David Spiegelhalter]

These plausible values, I think, are open to some criticism because they're predicted conditional on knowing the difficulty of the questions, as if these are fixed constants, even though they're only estimated based on a subsample of students and assuming that the difficulty is the same for all students around the whole world.

[Ruth Alexander]

So when you see the league table of countries, the first thing to understand is that each country has been ranked according to an estimate of national performance. And the model assumes that certain questions are the same level of difficulty, that a question asked to a child in England, say, is of exactly the same level of difficulty as the version of the same question asked of a child in Denmark. And this has been doubted by other statisticians. And back in 2011, we reported on this and put the criticism to the OECD's Deputy Director for Education, Andreas Schleicher, who oversees the PISA tests. He had this to.

[Andreas Schleicher]

I'm not saying that you cannot possibly find a task in which Denmark does significantly better than England and another task in which Denmark does worse than England. The question is, will your test as a whole appropriately compare British and Danish performance? And that is the question that we have demonstrated in every possible way. The model is always an approximation of reality. Any scaling model is an approximation of the reality that you observe. The question is not, does the model exactly fit the reality? The question is, does the model fit the reality such that there is no distortion of the results?

[Ruth Alexander]

So Andreas Schleicher stands 100% behind Pisa's statistical calculations. But he says that people shouldn't pay too much attention to the precise country rankings because there is a margin of error in the calculations. However, he says the broader test results do tell us something important about education.

[Charlotte MacDonald]

I think you're ready for the final question.

[Ruth Alexander]

Oh, yes. And I've seen this question. It's a level six question, the hardest, and it does require some thinking through.

[Charlotte MacDonald]

It does. So here we go. We're thinking about Helen and her bike again. Helen rode her bike from home to the river, which is four kilometres away. It took her nine minutes. She rode home using a shorter route of three kilometres. This only took her six minutes. What was Helen's average speed in kilometers per hour for the trip to the river and back?

[Ruth Alexander]

Well, she rode four kilometres in nine minutes, followed by three kilometres in six minutes. So she cycled for a total of 15 minutes. Fifteen minutes is a quarter of an hour. So I need to work out how far she'd travel in an hour. To do that, I multiply the distance she traveled in a quarter of an hour by four. She went seven kilometers in a quarter of an hour. So seven kilometres multiplied by four is 28 kilometers. Helen was cycling at 28 kilometres an hour.

[Charlotte MacDonald]

Correct. Now, about 30%, almost a third of 15 year olds in Shanghai manage that level of question. The average number, though achieved by countries such as France, Austria and the UK

[Ruth Alexander]

was just 3%, which does seem low. Should the low scoring countries react to this? Well, no, according to David Spiegelhalter. He says that while there's something in the fact that some Asian countries are consistently high performers, other countries should be wary of going the way of Germany, Denmark and Norway, who suffered what's become known as pisa shock in 2001. When the first test results didn't put them at the top of the world table as they'd expected, it led them to make changes to their education systems.

[David Spiegelhalter]

I quite like their maths test. I think it'd be great if kids could answer these maths questions. I think it'd be really useful for society and for their ability to cope in the world. But actually it's not the be all and end all and aligning policy along a single performance indicator or even one or two performance indicators can cause all sorts of distortions in the system. And I think what it should do is create a wider debate about what do you want out of an education system. But when you look at what's going on in Korea, which does very well, the kids are miserable 65 out of 65 in the league table for happiness in schools. The Japanese kids don't feel pleased when they get good grades.

[Ruth Alexander]

In the end, the critic comes to a view that's similar to the proponents. Doing an international comparison of education systems isn't a pointless exercise. Broad lessons can be learned. Just don't take the numbers too much to heart. If there are any numbers you'd like us to have a look at, do email us. Our address is more or [e-mail] and you can download this programme and previous editions at our website, bbcworldservice.com more or

[Charlotte MacDonald]

less

[BBC Announcer]

There are dozens of different podcasts now available from the BBC, including news, documentaries, science, business, arts and sports. The details of them all go to bbcworldservice.com podcasts.