
Investigating the public sector pay premium, statins and the 'decline effect'.
Loading summary
Hannah Barnes
Thank you for downloading the More or Less podcast from the BBC. You can find out more about our programme on our website, BBC.co.uk radio4. But before you do that, here's Tim Harford.
Tim Harford
Hello and welcome to More or Less, your faithful guide to the numbers in the news and in life. Obviously two very different things. This is the last programme in the series and we'll use it to ask,
Hannah Barnes
is modern science too complex to be
Jonah Lehrer
left to the scientists?
Tim Harford
We'll answer this.
Hannah Barnes
If everyone over 55 were given cholesterol and blood pressure lowering drugs, as researchers recently suggested, how many people would take those drugs unnecessarily?
Tim Harford
And we'll discuss this.
Jonah Lehrer
The effect sizes start out very big and very robust and they get smaller and smaller and smaller until after a few years there is no effect size left. The thing itself seems to have disappeared.
Tim Harford
But first, by popular demand, let's look at this week's news that public sector workers are paid 40 or even 43% more than those in the private sector. Many listeners tweeted and emailed about that number, which made headlines in several newspapers and even led the front page of the Daily Telegraph. I spoke to Antoine Bozio, a senior research economist at the Institute for Fiscal Studies who been studying the difference between pay in the public and private sectors. I read in the Daily Telegraph headline and I asked him if he thought it made sense.
Antoine Bozio
Not really. So where does this number come from? Well, this number comes from looking at two the average pay in the public sector and the average pay in the private sector and then pensions value in the public sector and the average pensions value in the private sector. The problem is that public and private sectors are two different sectors and you can't compare means averages in the two sectors because you're not comparing like for like. The public sector is composed a lot more of skilled workers than is the private sector. It will be exactly the same thing as to say, well, doctors are paid a higher wage than workers at Tesco and therefore it's unfair. So if you just compare the means in both sectors, you'll get a very big number. But this is largely represent the difference in composition between the two sectors.
Tim Harford
Antoine Pozio's point is that the typical public sector worker is older, more experienced and better qualified than the typical private sector worker. So wouldn't we expect him or her to be better paid? It's worth pointing out that even Bosio thinks there's a pay gap, but it's much smaller than the Telegraph's figure. After adjusting for composition, the public sector Pay premium seems to be about 7 or 8%, plus a more generous pension, which is certainly nothing to sneeze at, but it's not 43% either. The figure comes from a report published by Policy Exchange, a centre right think tank. Their report actually comes up with very similar figures to those calculated by the IFS for the public sector pay premium without pensions and adjusted for workforce composition. So perhaps they'd been misrepresented by the newspapers. Matthew Oakley is one of the authors of the Policy Exchange research. He had no complaints about the use of the figure for 43%.
Matthew Oakley
I think it's a fair reflection of one of the measures we put in the report. Yes, we published a range of measures in the report showing how the earnings differ and how that differs when you add in pensions as well. Obviously, as a lot of people will tell you, this kind of thing is really hard to measure. And the important thing for us was that on all the measures that we showed, actually that gap, the higher public sector pay showed up on all of those measures we used.
Wesley Stevenson
Well, I mean, it's fair to say that it was higher on all measures, but we reasonable estimates that I've seen go down into the single digits.
Matthew Oakley
The most conservative estimate we publish is, I think, 8.8%, obviously just below 10%. The lower numbers you've seen account for things like difference composition in the public and private sectors, between the age balance, qualifications in the public and private sector hours worked. All of those things are accounted for. Now, once you start accounting for those things, you can actually account for too much. I think the IFS has said themselves that things like qualifications aren't a particularly good measure of productivity. And actually so when you're controlling for these things, you're not actually too sure what you're doing. And so despite the kind of fact you're trying to control for them, nobody really knows what the best measure is. And I think, as I said before, the key point is that the gap exists no matter what measure you use.
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
Do you think it's helpful to publish
Wesley Stevenson
a figure as high as 43%, accepting that there is no perfect way to compare like with like. There's no perfect way to compare doctors with workers in Tesco's or managing directors of companies with leaders of city councils? Still, is it helpful to put out a figure that makes no attempt to do that? And of course, what happens is the newspapers lead with this figure.
Matthew Oakley
We were quite clear what the figure represented and we said ourselves that this wasn't taking into account the differences between the public and Private sectors, obviously, we can't choose which figures the press will publish.
Wesley Stevenson
You can't choose which figures the press will publish. But Neil o' Brien of Policy Exchange has defended the use of that figure. Maybe you should just have published the figure that you thought was the very best.
Matthew Oakley
As I've just said, I don't think there really is a figure which is really the best. If that figure was available, the trade unions and other commentators would be using it.
Tim Harford
You've called attention to the trend and
Wesley Stevenson
you pointed out that according to your calculations, the public sector premium over private sector workers has pretty much doubled over the last couple of years.
Matthew Oakley
Yep.
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
Is it really fair to focus on just two years?
Wesley Stevenson
I mean, we're talking about the recession. We know the private sector has been hard hit. We know the Government was stimulating the economy. We know there's now a pay freeze. I mean, is two years really a representative timescale to look at?
Matthew Oakley
I think it's completely representative. What we're trying to say is that there's a clear choice here between the public sector getting paid more and the gap between the public sector and the private sector growing and the number of jobs which the public sector is likely to lose. We're arguing that if the public sector unions and the public sector had shown more pay restraint, there might not be such a need for so many job losses.
Tim Harford
Matthew Oakley of Policy Exchange. As you might know, when I'm not making more or less, I'm usually to be found in my office in Trumpetshire county hall, where I'm treasurer. Trumpetonshire's a peaceful place, made famous, of course, by an early attempt at reality television. But Trumpetonshire is not immune from the Government's deficit reduction programme. And I've already cut the Trumpton fire brigade by 14.28571% through the simple expediency of sacking fireman dibble. But on the advice of Policy Exchange, I've decided to go further.
Fireman Dibble
Trumpeton Fire Station.
Tim Harford
Flack. Hello, it's Tim Harford here. All well?
Fireman Dibble
Unfortunately, I'm extremely busy this morning.
Tim Harford
Well, we're all busy, Captain. How are the men?
Fireman Dibble
Oh, they have their ups and downs like the rest of us.
Wesley Stevenson
Dear, dear.
Tim Harford
Look, I'm calling about your remuneration package. How would you describe it?
Fireman Dibble
Very attractive.
Tim Harford
Well, so would I, and that's what's worrying me. Flack, your pay's outstripping the private sector. I'm going to have to freeze it for two years.
Fireman Dibble
What did you say?
Tim Harford
You heard me. And the same goes for the boys.
Fireman Dibble
Just wait till I tell the mayor.
Tim Harford
Don't waste your time, Captain. The mayor's on board.
Wesley Stevenson
Sorry about that.
Tim Harford
You're listening to More or Less, in association with the Open University and I'm Tim Harford. Now a scientific mystery story.
Jonah Lehrer
Many scientific studies in a wide range of fields from psychology to psychiatry to different kind of clinical trials. They all exhibit the same puzzling phenomenon, which is they started looking very robust. You know, someone publishes a peer reviewed study and it gets lots of attention. It's in a, you know, high quality journal. And then something happens. The results start to fall apart. Where what you see is the effect sizes start out very big and very robust and they get smaller and smaller and smaller until after a few years there is no effect size left. The thing itself seems to have disappeared.
Tim Harford
That's the voice of Jonah Lehrer, a columnist for Wired magazine and the author of How We Decide. Jonah wrote an article for the New Yorker titled the Truth Wears Off. Is there Something Wrong with a Scientific Method? I asked him to tell us more.
Jonah Lehrer
The article begins by looking at second generation antipsychotics. It's one of the most profitable segments of the drug market. These are drugs given to people with schizophrenia and bipolar disorder. And when you go back 15 years ago, second generation antipsychotics as a category, they seemed like a miracle cure. They seemed far more effective than first generation antipsychotics. We just had these drug companies justified, charging up to 10 times as much money per pill. And then the decline effect comes into play. Now we're at the point where you can look at the big studies done by the National Institutes of Mental Health here in America. And second generation antipsychotics as a category no longer seem any more effective than first generation antipsychotics. So now you have these big government agencies saying it's hard for us to justify paying 10 times as much per pill. It doesn't have fewer side effects and doesn't cure our symptoms any better than the old drugs and the 50 year old drugs. I do think we have to be aware that when we probe anything, first of all, it's very, very difficult. That's the first reason the decline effect exists. It's just because it's hard to understand reality. It's really complicated and there are all sorts of causal forces at work. There are many more variables interacting than we can even begin to imagine. So it's very tough to do a well done, controlled experiment that gets at what you want to get at. The second reason the decline effect exists is because scientists are human beings.
Darrell Ince
Too.
Jonah Lehrer
I think we've got this very antiquated model of scientists as objective robots, you know, that they just pursue the truth, whatever it is. But of course, scientists are human beings, and human beings have passions and egos that we've got this annoying tendency to see what we want to see and to disregard the rest. So the decline effect doesn't mean that all of our research is equal to research into parapsychology. It doesn't mean that reality is unknowable. It doesn't mean that we should all be postmodernists. It simply means that science is hard and scientists are human beings.
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
To what extent can we sort this out with better procedures? It seems to me that one of the possible explanations contributing to the decline effect would be publication bias. Interesting. Research is more likely to get published, and then when you try and replicate it, maybe some of the interest was just chance. So if we had a better system for dealing with publication bias, do you think that we would just get rid of this problem, or at least some of this problem?
Jonah Lehrer
I think the evidence suggests that publication bias plays a big role. One of the best points of evidence comes from work on fluctuating asymmetry. Fluctuating asymmetry was basically this notion that when females, especially female birds, are trying to figure out who to mate with, they seem to prefer mates who are more symmetrical. And that's because symmetry is a proxy for genetic fitness. So if you're asymmetrical, say if your wing feathers don't line up properly, that probably means your genes aren't of a very high quality. So those female birds don't want to have your babies. So it was found first in the early 90s, and then it was confirmed several times. So if you look at the early studies on fluctuating asymmetry of the first 10, eight confirmed it. So it seemed like a very robust idea. It was published in, you know, Nature, in Science and all the big journals. And then this troubling thing starts to happen where, if you look at, say, five years out now, there are 13 different tests of fluctuating asymmetry, and only two can confirm it. The effect size is also basically zero at this point. So in a matter of five years, this phenomenon seems to have disappeared. And when you start asking scientists what happened, the story they tell is that if you wanted to get published in a big journal, you needed to show that fluctuating asymmetry happened. But then after a few years, what happened is the incentives of the journal start to shift. Now, all of a sudden, it's more noteworthy to disprove it, to falsify it. And so now it's much easier to get those papers in. Perhaps the most troublesome element of publication bias is the way it reinforces and accelerates these fads and trends within science. So I think to start with, publication bias may be somewhat inevitable just because, you know, we're always going to have these scientific journals, we're always going to have peer review. But I do think we can do a better job of making sure that journals aren't just engaged in these fashions and trends.
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
I've been interested in the response to your article in the New Yorker. I was at a meeting of fairly high profile politicians and civil servants involved in research and science funding here in the uk and your article got very high praise and it also got extremely robust criticism. Some people were very angry with what you wrote and the idea that you'd really over egged it and told a story that the scientific method was basically dead and we didn't know why. Whereas in fact, what you should have done is explain there are certain problems and we need to fix them. So do you plead guilty to that charge?
Sir Nicholas Wald
No.
Jonah Lehrer
This was a very difficult article for me to write. On the one hand, the last thing we need is more just naive, stupid skepticism of science. You know, in America, the majority of people don't believe in natural selection. You know, they don't believe in Darwin, they don't believe in evolution. You have an epidemic of people not giving their kids vaccines. We hold all sorts of pseudo scientific beliefs. The last thing I wanted to do was to give people an excuse to not believe in science. But in the end, the, the reason I wrote the article was because I do think there are real problems with how science is done in the 21st century. These aren't just abstract intellectual problems involving how female birds choose male birds. These are problems that really affect our lives. You have people like John Ioannidis, who, you know, he does these vast studies of clinical trials that are out there. He just looks at top quality journals, then within that he looks at the most heavily cited papers from those journals and what he finds is that somewhere between 40 and 50% of these gold standard trials, they either show significantly weaker effect sizes or outright contradicted within 10 years. So we should be troubled by it. We should be concerned that here are the studies we pay the most attention to, they change medical practice and nearly half of them are somewhat contradicted within 10 years. So something is not right. We are not doing the best we can. I think there are obvious flaws in the system that can be fixed. So, you know, I wrote the article to call attention to these problems.
Tim Harford
Jonah Lehrer, author of How We Decide and we'll be coming back to the scientific method and the difficulties in making it work as well in reality as it does in principle in a few minutes. Now, several of you tweeted me. I'm Tim Harford on Twitter or emailed more or lessbc.co.uk to ask what we thought about last week's story on the mass prescription of drugs to fight heart disease. The story began with Professor Sir Nicholas Wold. He's the inventor of the Polypill, a combination of drugs such as statins and blood pressure lowering medication. And last week he argued that we should be offering medication to everyone over 55 in order to bring down the numbers of people having heart attacks and strokes. At the moment, we screen people for risk on the basis of their cholesterol levels, whether they smoke and how high their blood pressure is before prescribing such drugs. But Professor Wald thinks we can forget the screening and just use age as the criteria for offering the medication. As part of his argument on the Today programme, he said this about the treatment.
Sir Nicholas Wald
We know that screening for cardiovascular disease and offering statins and blood pressure can reduce the risk by over 50%. There are some side effects that are bothersome. So not everyone on this could stay on. They would have to see their doctor for adjustment. And it is not a totally risk free regimen, it's just that the risks of harm are very small in relation to the gains in the reduction of heart attacks and strokes.
Wesley Stevenson
Fair enough, but that's only half the story.
Tim Harford
Wesley Stevenson's here. That 50% reduction in the risk of
Wesley Stevenson
having a heart attack or a stroke.
Tim Harford
It sounds like a big number, Wes, but is it?
Dr. Carl Hennigan
Well, it lacks a bit of context. 50% of what? Exactly? So this is a question that I put to Dr. Carl Hennigan, Director of the Centre for Evidence Based Medicine at Oxford University, and also a GP.
Dr. Carl Hennigan (continued)
If you take a 55 year old, a 55 year old, if you take men and women together, the risk is about four and a half to five percent in the next 10 years. So that's about five per hundred. So actually once you suddenly say a 50% reduction, you go from about five in 100 to about two and a half to three people in 100. Suddenly you're only saving about two or three events in every 100 people treated. And so anytime anybody gives you the relative effect, what you need to do is stop, say, can I have the baseline measure? What's my risk today? And actually then think about how much reduction do I get.
Dr. Carl Hennigan
It's also important to note that this is a 50% reduction on taking nothing at all. So offering more people the drug won't see a 50% reduction in the current levels of heart attacks and stroke in this country because currently we are treating large numbers of people.
Wesley Stevenson
Ok, so three events are prevented per hundred people.
Tim Harford
Most people would be pretty pleased. Isn't treating 97 people who don't benefit
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
a price worth paying?
Dr. Carl Hennigan
Well, yes, assuming that the risk of side effects is low. I spoke to Sir Nicholas Wald and he says that the evidence shows that the benefits do outweigh the risks. And that's what he said earlier. But others, including Dr. Hennigan, say in order to judge this, we have to know more about the risks of side effects and he thinks we need to do more work in order to judge this.
Tim Harford
Well, thanks, Wes. You're listening to More or Less where we're always grateful for your emails. Our address is more or lessbc.co.uk. sometimes we receive questions we simply can't answer.
Wesley Stevenson
A number of people, for example, have
Tim Harford
asked us how many civilians have died violently in Iraq since 2003. We don't know. So we asked Kelly Greenhill from Tufts and Harvard universities, whose book Sex, Drugs and Body Counts deals with the use and misuse of conflict statistics. Kelly says it is possible to hazard a rough guess. She began by outlining the methods used to produce these statistics.
Kelly Greenhill
There are principally two techniques. One is known as passive surveillance and the other involves a more active approach. And those are survey based studies. Passive surveillance techniques entail cross referencing data provided in media reports with data coming out of governmental sources from NGOs and from a variety of other actors on the ground. But some criticisms associated with passive surveillance is that they can also lead to undercounting due to, among other things, underreporting.
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
And what about the higher end, the survey method?
Kelly Greenhill
The figures that have come out of the survey based studies have varied tremendously from several hundred thousand to approaching one and a half million. The study that has passed muster best is the study conducted by the World Health Organization that measured deaths between 2003 up through the middle of 2006.
Wesley Stevenson
And what was their method?
Kelly Greenhill
The bottom line is the idea is to be out there in the field interviewing people on the ground, asking them whether they've lost any family members or whether that's immediate family or more extended family. Inquiring how those people died, in many cases asking for some verification, death certificates or other documentation to try again to avoid overcounting.
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
So what are the problems with all these kind of surveys?
Wesley Stevenson
And presumably partly it's just dangerous to wander around Iraq asking questions.
Kelly Greenhill
Yes, simply trying to gather good conflict data in the context of an ongoing war is a hazardous and risky proposition. At. In the case of Iraq specifically, there have been some problems that I might think of as sort of apples versus oranges data acquisition. But apples versus apples reporting and replication of estimates. What gets counted also varies. So some studies include combatants, others only civilians, non combatants, some count combatants and law enforcement personnel and so on and so forth. So basically, who is getting counted has varied widely.
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
What is the range that you think is real?
Kelly Greenhill
We could reasonably narrow the range of violent direct civilian deaths since the invasion to somewhere between about 120, sort of 125 to about 400,000, plus or minus, which, to be clear, is still a very significant range and still very much an estimate.
Wesley Stevenson
And still a very large number.
Kelly Greenhill
Yes, still a very large number.
Tim Harford
Kelly Greenhill, the author of Sex, Drugs and Body Counts and Kelly's, written a more detailed essay for us, which you can find on our website, BBC.co.uk more or less now, is modern science simply too complex to be left to the scientists? In 2006, a group of researchers at Duke University in America published what looked like a major breakthrough in allowing more personalized, less debilitating chemotherapy. But two medical statisticians, Keith Baggeley and Kevin Coombs of the University of Texas, reckon they'd found serious problems with the original research. Yet it took them years to bring their concerns to wider attention, by which time expensive and intrusive clinical trials were underway. Eventually, much of the original research was retracted and one of the Duke researchers accepted responsibility and resigned. But what broader lessons can we draw from the affair? I spoke to Darrell Ince, professor of computing at the Open University. Professor Ince recounted this sorry tale in last week's Times Higher Education Supplement. He says part of the problem is the journals aren't keen on publishing negative research which criticises someone else's errors.
Darrell Ince
Bagley approached the editor of a biological journal and the editor said, well, I'm sorry, but it's really a very negative paper. There's this thing about research journals that it's very exciting to announce new results. It's somewhat embarrassing and less exciting to. To deal with results which don't hold, where you have to retract the papers.
Wesley Stevenson
I'm no Philosopher of science here, but I thought the whole idea of science was that you put theory and evidence out there and people try and falsify it. It seems a bit strange to say, well, the papers have been negative, we can't publish it.
Darrell Ince
Falsifiability or deniability is key to the scientific process. And I think journals should start looking at how they deal with this issue.
Wesley Stevenson
You talk about bioinformatic forensics, the idea that it takes months and months and months to sort of pull apart the data. To a lot of people, that's going to sound strange. I think a lot of people will imagine there's a research paper. You read the research paper, you check the sums, and you know it's either right or wrong.
Interviewer/Host (possibly Tim Harford or Wesley Stevenson)
But I mean, from what you're describing,
Wesley Stevenson
it's actually a lot more complex than that.
Darrell Ince
Well, it used to be very simple. You can see exactly what the research is about, where it led, what they excluded, what they included. Now, if you come to the present day, where you have tons of data, lots of software, maybe scientists who are not practiced in writing computer programs, you have problems. And one of the problems that arises is that it's awfully difficult sometimes to get hold of data and to get hold of computer programs. There was a key study on something called microarray technology. This was the technology that was used by the Duke University people. And the study looked at, I think, 18 research papers to see whether they could be reproduced. And what they discovered was that only two of them can be reproduced.
Tim Harford
It seems extraordinary.
Wesley Stevenson
Forgive my naivety, but surely when you publish a paper, surely all the data and the software coding you use to process the data, surely that should all be available.
Darrell Ince
Well, it should and it is actually changing. There are many journals, though, that are a distance away from that. Some journals live in another age where researchers were very collegiate and where they swap data quite easily, etc. But now we live in an age where universities are becoming commercial entities. What they need to do, I think, is to just move along a bit to the point where data and code is made available and you don't get the publication of your paper unless that's lodged somewhere.
Wesley Stevenson
Are you saying also that scientists are just not quite statistically and technically competent enough to process data that these modern research programs are generating?
Darrell Ince
Well, I don't want a bad mouth, because scientists are highly, highly skilled people, and to expect them to be really good at their own discipline and be able to program a computer and be able to handle the sort of advanced statistics that are needed is too much. There'll be projects where you just don't generate a lot of data, where the stats that you learned in your undergraduate days are fine small scale experiments. But when you're getting to the point where you've got machines which generate gigabytes of data, where there's error in the data, where you get what's called a batch effect, where one run of the machine differs from another run of the machine, then you do need somebody along who can say, well, just stop for a minute and just. Let's think about the issues about the accuracy of the data.
Tim Harford
Professor Darrell Ince of the Open University. Now, as I've mentioned earlier in the series, I was forced to sack Dibble from the Trumpton fire brigade as a cost saving measure. I expected a backlash, but to be honest, the strength of feeling has taken me completely by surprise. Since Robert Peston launched his campaign to reinstate the Trumpton one, everyone's jumped on the bandwagon, from Windy Miller to Mrs. Minton. Don't get me wrong, there can be no turning back. I inherited an economic mess and it's my job to sort it out. But as the series draws to a close, I want to put things right. I want Dibble to know it wasn't personal and that I wish him well. So if you'll excuse me just one moment.
Fireman Dibble
Hello?
Wesley Stevenson
Is that Dibble?
Fireman Dibble
Yes, it is.
Tim Harford
Dibble, it's Tim Harford. Oh, Dibble, I know I'm not your
Wesley Stevenson
favorite guy right now, but I just wanted to say I'm sorry I had to let you go.
Tim Harford
Really, Dibble, it wasn't personal. I wanted to call and tell you that.
Fireman Dibble
A charming gesture.
Wesley Stevenson
I'm sorry.
Tim Harford
Is there any way we can be friends?
Fireman Dibble
If you will step into my. I will go and plaster your for you now. How's that?
Tim Harford
Hmm. That didn't go so well. Perhaps it's all for the best that the series ends here, but we'll be back in August, so why not subscribe to our podcast via BBC.co.uk more or less to make sure you don't miss us when we come back. And please keep your comments, questions and suggestions coming in to more or lessbc.co.uk. but for the present series, goodbye.
Hannah Barnes
More or Less is presented by Tim Harford, the Financial Times undercover economist. The More or Less team is Richard Varden, Richard Knight, Wesley Stevenson and me, Hannah Barnes. More or Less is made in association with the Open University.
Fireman Dibble
Pew Pew, Barney McGrew, Tuffett Grub.
More or Less (BBC Radio 4) – “Public Sector Pay”
Date: May 13, 2011
Host: Tim Harford
Episode Theme:
A critical look at public sector pay statistics, including the recent news that public sector workers are allegedly paid 40-43% more than those in the private sector. The episode also explores the limitations of modern scientific research, medical statistics around the “polypill”, and the challenges of measuring casualties in Iraq.
This episode examines contentious statistics about public sector pay premiums, dissects misleading uses of averages in comparing public versus private pay, and provides insight into scientific reproducibility and medical statistics. It weaves interviews with experts, case studies, and light-hearted sketches to illuminate how numbers in the news can be distorted or misunderstood.
This episode of "More or Less" highlights the pitfalls of headline statistics without proper context, shows how numbers can be used to mislead through selective interpretation, and declares the need for skepticism and methodological rigor — not just in economics and public pay debates, but across science and medicine as well.
For further reading and detailed essays, listeners are invited to visit the program's website, and send in questions and comments.