Survivor bias and cervical cancer audit: all is not what it seems

In todays post we discover how lessons learnt during World War two can be applied to cervical cancer audit.

The Curious Case of the Missing Bullet Holes

During the Second World War the Americans set up a group of applied mathematicians to see whether they could solve problems related to the logistics of war. One such problem was where to place protective armour on aeroplanes. They couldn’t just reinforce them everywhere – they’d be too heavy to fly. They asked the statistician Abraham Wald, a refugee from Nazi Europe, for advice. He told them to plot on an outline of an aircraft the position of all the bullet holes on fighter planes and bombers. The resulting “graph” was a revelation. The wings and most of the fuselage appeared riddled with bullets. The front of the plane was relatively free of damage. The answer seemed clear – add armour to the wings and fuselage. But Abraham Wald told the generals that that was precisely the wrong thing to do. There was no reason to believe that the Germans could aim precisely at one part of the plane. The bullets would strike at random across the whole flying machine. And the generals knew that the front of the plane was not more resistant to bullets. No, the issue was that nobody really cared about the holes in the planes that made it back to base. The real problem was the planes that blew up or crashed. The generals had no idea where they were damaged. But Abraham Wald did. He argued that they must have been damaged precisely in the places where we never observe damage: in the cockpit killing the pilot; in the engine causing the plane to crash; in the fuel tank causing an explosion. When you hear it, Wald’s analysis is obviously right.

So, what has all this got to do with cervical cancer?

The lesson of the aeroplane story is called survivor bias. It is defined as the error of concentrating on things that made it past some selection process and overlooking those that did not. When studying cancer screening, the survivor is not the patient but the cancer! Cancer screening (the cancer’s enemy) is trying to identify and destroy pre-cancer. Cancers are only diagnosed when screening fails to prevent them from progressing. In a cervical cancer audit it is important to remember that we are only studying those cancers that were not prevented by screening.

Consider what happens to a woman with a positive result (severe dyskaryosis, say) on cervical cytology. She might already have cancer; in which case it is probably diagnosed very shortly after screening. She might have pre-cancer in which case she is hopefully treated, but there is a small chance that she will not go to colposcopy or that she will develop cancer despite going for colposcopy. Or her cytology result might have been a false-positive and after further investigation (colposcopy) she is not treated and will not develop cervical cancer in the short term.

Cervical cancer audits

Now consider what happens to women with a false negative cytology that on review is considered positive. We can assume that they are very rare, but if we only review slides in women who have cancer, we will have enriched the sample for such false negatives. If we exclude the screen-detected cancers, then the only positives (on review) will either be slides that were originally false-negatives or women who have developed cancer despite being referred to colposcopy. In audits of “normal” slides from women who subsequently developed cervical cancer it is hardly surprising that a high proportion will be found on review to have had missed abnormalities.

HPV Negative Cancers

As readers of this blog will know, I am a great fan of HPV primary screening. I am confident that its introduction later this year will lead to a noticeable reduction in cervical cancer – from 2022 we should see about 500 fewer cancers a year in women aged 25-69 in the UK. However, like all tests it is not perfect and there will be some women who develop cervical cancer despite having a negative HPV test. In one study from America in a population of women being screened by both HPV testing and cytology, there were 81 cases of cervical cancer diagnosed over a seven year period. 31% of those cancers were in women who tested negative on their first HPV test. Some people have argued that these results should be a warning against switching to HPV testing. But I think it is simply a case of survivor bias.

Let’s consider what would happen if we screened myriads of women by both HPV testing and by cytology. Suppose that among those women 1000 would develop cervical cancer over the next 6 years without some sort of intervention (i.e. treatment). The results might be as shown in the table below.

Overall HPV testing would miss just 70 (7%) of those precancers compared. But suppose we have a screening programme that is based in cytology, so the only cancers that we observe are in women with normal cytology. Then we will only observe 234 cancer, but among those 234 cancers 47 (20%) will have been in women with a negative HPV test. In truth HPV testing would detect 93% of pre-cancers, but the effect of survivor bias when looking only at the cancers that develop on women who have been screened by cytology is to make it look as if HPV would only detect 80%.

Looking at the table one can see that introducing HPV primary testing may lead to us missing 23 pre-cancers that would have been picked up by cytology, but it will lead to the detection of some 187 pre-cancers that are currently missed by cytology. Of course, we will never know about those 187 precancers – thanks to screening they will never become cancer. But if we investigate the cancers that develop despite screening we should not be surprised if most of them were HPV negative just a few years before the cancer was diagnosed.

Thank goodness Abraham Wald has shown how to see through the holes.

The views expressed are those of the author. Posting of the blog does not signify that the Cancer Prevention Group endorse those views or opinions.

Share this page

Subscribe to our mailing list to get updates of new posts.