Medical research and the common law of confidentiality

In this series, members of our group, the wider research field & individuals from public interest groups will debate the use of patient data in medical research. We encourage you to get involved in the comments and let us know your thoughts!

We begin with a letter, given by Julian Peto FRS of the London School of Hygiene and Tropical Medicine to the Lords Science and Technology Committee’s inquiry into Ageing: Science, Technology and Healthy Living on the 25th of February 2020.

Professor Peto wrote:

In my field of research, cancer epidemiology, we now spend more than half our time acquiring or renewing permissions for data access, abstraction or matching to link specific groups of people (e.g. women who did not attend for cervical screening, workers in asbestos factories in the 1950s, or people in old clinical trials) to their subsequent medical records. 30 years ago cancer registrations and deaths were immediately available at minimal cost through the Office of National Statistics once a study had been approved by a medical research ethics committee. NHS data on current systems also include hospital and GP records, but access is delayed, often by years, because bureaucrats who manage NHS databases now adjudicate our requests for data against a background of frequent changes in their regulations and approval processes. Most GP records cannot be accessed at all because they are held on commercial systems on behalf of the GP practices.

I began working as a medical statistician in 1969, and until about 1990 we had free access to named patient records in most of our studies. In 1972 I was updating and analysing the MRC Tuberculosis (TB) Unit’s cohort of TB patients diagnosed before 1957 to see if isoniazid is carcinogenic in humans (Stott et al 1976), so I was browsing the old TB case notes in the Frimley hospital basement with the permission of Reg Bignall, a senior consultant. Frimley had been a TB hospital, and Reg had conducted randomized trials of one, two then three drugs there in the early 1950s, reducing pre-war mortality of more than a quarter to almost zero. In the notes of one young man, I found Reg’s excited scrawl describing the most rapid clearance of a TB lung X-ray he’d ever seen during treatment. That sort of experience is educational as well as enjoyable for a young statistician. Today Reg would be a criminal for giving me permission to be there unsupervised. Research should still be governed by the Medical Research Council guidelines before the first Data Protection Act, which stated that bona fide non-commercial medical researchers could have access to relevant patient records. All studies must of course be approved by expert medical research ethics committees, which make reasonable and often valuable recommendations on our protocols. Commercial researchers should be separately regulated to ensure that their confidentiality and data-sharing arrangements, as well as their aims, are in the public interest.

The regularly updated MRC guidelines on medical research are no longer based on the assumption that medical researchers are trustworthy. The common law duty of confidentiality is now defined as the duty of confidence “when you know information about an identifiable individual and they have a reasonable expectation of privacy with respect to that information (e.g. patient and doctor)……When consent is not possible or is impractical, the law allows disclosure in certain circumstances.  In England and Wales, Section 251 of the NHS Act 2006 and subsequent Regulations … allows for the Common Law Duty of Confidentiality to be set aside temporarily for defined medical purposes.  This allows time-limited disclosure of ‘confidential patient information’, without patient consent, for medical research.  Section 251 should only be considered as a last resort, when all other options have been exhausted” (MRC 2017).

This obsession with personal privacy was initiated by a vocal minority of professional ethicists. An egregious early example of their power to invent and enforce novel ethical principles was the assertion in 1987 by an expert witness to a Select Committee that a test must confer some benefit on the patient. This delayed routine anonymous HIV testing of discarded blood samples to monitor this new epidemic by several years (Black et al 1987). We can no longer hold any identifying information on most of the people we study, leading to a marked deterioration in the reliability of any record linkage, particularly for follow-up of retrospective patient or occupational cohorts. Enforced eventual data destruction prevents potentially valuable longer-term study of such groups. The worst effect, however, is that many British epidemiologists spend more than half their time on this bureaucracy, exchanging hundreds of emails with civil servants and the various companies that now manage NHS databases before a study already approved by an ethics committee can even begin. Extending data access to update our studies incurs high annual charges and further bureaucracy. Even the routine transfer of data between Public Health England and NHS Digital was recently suspended while they reconciled their revised data transfer permissions and procedures.

This is a spectacular example of how mumbo-jumbo is conquering the world in science as in other fields (Wheen 2004). There was never popular support for this assault on British medical research in the name of personal privacy when the value and harmlessness of medical research requiring patient records had been explained before asking the question (Peto et al 2004). During 50 years as a medical statistician I held hundreds of thousands of named personal records on computer files, and I have never heard of a breach of privacy of any importance by anyone in my position. Today there is a leak of millions of personal records every month or two from a major social, financial or police database, but still none from medical statisticians. Our burgeoning army of data custodians and regulators was created to solve a problem that didn’t exist.

The legal origin of this blossoming of regulations was the first (2001) Health and Social Care Act, which created the Patient Information Advisory Group (PIAG, now renamed the Confidentiality Advisory Group, or CAG) with authority to permit (but not instruct) data controllers to make specified personal records available for research. Lord Falconer (Solicitor General then Lord Chancellor) stated that access to medical records for research was unaffected by the Data Protection Act and entirely determined by the “common law of confidentiality” that his Government had (probably unwittingly) redefined by referring to it in the 2001 Act (Peto et al 2004). By authorizing PIAG to overrule the common law of confidentiality the Act implied that the common law (i.e. accepted practice) dictates privacy of medical records even from medical researchers. For centuries common practice and hence common law had been the opposite.

This unnecessary (and offensive) attack on our integrity as well as our work could be reversed by minor amendment of the National Health Service Act 2006. CAG should have more rational terms of reference and wider powers, notably the authority to direct an agency holding NHS data (the data controller) to provide the data required for an approved study in the form requested. The responsibility for deciding whether it would be reasonably practicable to conduct the research without the requested data, and what editing is required to ensure that individuals will not be identified unnecessarily, should thus rest entirely with CAG. NHS Digital, Public Health England and other NHS agencies should no longer be permitted to adjudicate what data they provide and in what form, and would thus bear no legal responsibility. Records are already indexed by NHS number so this would provide a single portal for accessing NHS records at no extra cost, an essential requisite for efficient long-term follow-up of the very large randomized trials needed to discover how to improve physical health and prevent dementia in old age (Collins et al 2020).

An important benefit of Brexit is the UK’s right to enact such an amendment irrespective of the EU General Data Protection Regulation. This would greatly enhance the productivity of large parts of British medical research while cutting costs. Restoring a beneficial convention that was universally accepted from Hippocrates until my middle age would, I believe, also prove popular.

What do you think of Professor Peto’s argument? Let us know!


The views expressed are those of the author. Posting of the blog does not signify that the Cancer Prevention Group endorse those views or opinions.

Share this Page

Subscribe to our mailing list

Be the first to comment

Leave a Reply

Your email address will not be published.