UK Statistical System Code of Practice Office for Statistics Regulation
UK Statistics AuthorityCorrespondenceDepartment of Health and Social Care

Letter to producers of health and social care statistics in England – OSR Covid Lessons Learned report

Dear Colleagues,

Today, the Office for Statistics Regulation (OSR) has published its review; Improving health and social care statistics: lessons learned from the COVID-19 pandemic. I am writing to senior officials in bodies responsible for health and social care statistics in England to ask for your support in implementing the recommendations from this review.

The COVID-19 pandemic resulted in a huge public appetite for data and statistics. You should be proud of the remarkable efforts of analysts in your organisations to meet this demand, overcoming challenges which would previously have seemed insurmountable.

The pandemic has also drawn attention to existing challenges for health and social care statistics. I am sure you would agree that the health system needs both to learn lessons and to build on its achievements in this area.

In England the number of organisations responsible for the production and publication of health data and statistics creates additional complexity. Strong leadership and collaboration are required across these organisations, so that shared priorities can be identified, and publication plans can be coordinated. This will create a clearer and more coherent picture for users and enable the system to be more responsive to emerging user needs. As senior leaders your support for the work of analysts in your organisation and across the health system will be vital in achieving this.

I have copied this letter to Lucy Vickers, Head of Profession for Statistics, Department of Health and Social Care; Chris Roebuck, Chief Statistician, NHS Digital; Mark Svenson, Head of Operational Information and Head of Profession for Statistics, NHS England and Improvement; and Emma Rourke, Director of Health Analysis & Pandemic Insight, Office for National Statistics.

Yours sincerely,

Sir David Norgrove

This letter was sent to the following:

  • Chris Wormald, Permanent Secretary, Department of Health and Social Care
  • Simon Bolton, Chief Executive, NHS Digital
  • Amanda Pritchard, Chief Executive, NHS England
  • Sir Ian Diamond, National Statistician, Office for National Statistics
  • Jenny Harries, Chief Executive, UK Health Security Agency

Matt Hancock response to Sir David Norgrove

Dear David,

Thank you for your letter of 2nd June setting out your concerns with our presentation of statistics on testing. We spoke later that day and I understand you have also spoken to Baroness Harding who leads the Test and Trace programme.

The testing programme has been set up at great speed over the last two months in the face of enormous urgent need. While standing up the programme and delivering the substance necessary, we have also sought to produce a range of statistics, in a transparent and open way, to allow people to understand what is being delivered. We are inevitably having to work through some challenging issues as the data systems behind these statistics are developed at great pace. I said in my first letter that I am keen for us to work closely with your team on developing the presentation on the range of information we publish.

Going forward I propose two outputs:

• First NHS Test and Trace will be publishing a weekly release which will set out core metrics on its operational performance. This will focus on the volumes of people covered by the programme, the speed of the user journey through the programme (e.g. test turnaround times) and overall outcome measures. I know that the NHS Test and Trace has been and will continue to engage with your team and the ONS in developing this publication.
• Separately to set out our understanding of the progress of the pandemic through the community my Department is working with ONS, PHE and others to present the best estimates of prevalence, incidence rates and antibodies on an ongoing basis in a co-ordinated way.

ONS and DHSC will work together to ensure these bulletins are coherent and aligned and present a clear overview of the progress of testing and the progress of the pandemic.

While we continue to publish daily updates on testing numbers. I’ve asked my officials to continue to work with you on improving the presentation of these. I have asked the Department’s Head of Profession for Statistics – Stephen Balchin – to write to the Office for Statistics Regulation with periodic updates on the specific points in your letter.

Yours ever,
Matt Hancock

 

Related Links

David Norgrove to Matt Hancock (June 2020)

Sir David Norgrove response to Matt Hancock regarding the Government’s COVID-19 testing data

Dear Secretary of State,

Thank you for your letter of 27 May, in which you described some welcome, though limited, additions to the official data on COVID-19 tests, including a proposed note on methods (not yet published at the time of writing). I am afraid though that the figures are still far from complete and comprehensible.

Statistics on testing perhaps serve two main purposes.

The first is to help us understand the epidemic, alongside the ONS survey, showing us how many people are infected, or not, and their relevant characteristics.

The second purpose is to help manage the test programme, to ensure there are enough tests, that they are carried out or sent where they are needed and that they are being used as effectively as possible. The data should tell the public how effectively the testing programme is being managed.

The way the data are analysed and presented currently gives them limited value for the first purpose. The aim seems to be to show the largest possible number of tests, even at the expense of understanding. It is also hard to believe the statistics work to support the testing programme itself. The statistics and analysis serve neither purpose well.

To mention just a few issues in relation to the data as currently presented:

  • the headline total of tests adds together tests carried out with tests posted out. This distinction is too often elided during the presentation at the daily press conference, where the relevant figure may misleadingly be described simply as the number of tests carried out. There are no data on how many of the tests posted out are in fact then successfully completed. The slides used in the daily press conference do not show the date when the tests were carried out;
  • the notes to the daily slides rightly say that some people may be tested more than once and it has been widely reported that swabs carried out simultaneously on a single patient are counted as multiple tests. But it is not clear from the published data how often that is the case. Figures for the overall number of people being tested have previously been published but are not available in the published time series;
  • the top summary presents the number of positive results from diagnostic tests (pillars 1 and 2) alongside the total number of tests across all pillars. This presentation gives an artificially low impression of the proportion of tests returning a positive diagnosis;
  • more generally the testing figures are presented in a way that is difficult to understand. Many of the key numbers make little sense without recourse to the technical notes which are themselves sometimes hard to follow. This includes the supporting spreadsheets, which, while welcome, make it difficult to extract even basic trends.

With regard to new data that are not currently made available:

  • test results should include for example key types of employment (e.g. medical staff, care staff), age, sex and location (by geography and place, such as care homes). How many people in what circumstances are infected? Where do they live?
  • for Test and Trace it is important that a statement of the key metrics to measure its success should be developed systematically, and published, to avoid the situation that has arisen in relation to the testing programme. The statistics will need to be capable of being related to the wider testing data and readily understood by the public, through for example population adjusted maps of hotspots.

I warmly welcome of course your support for the Code of Practice for Statistics. But the testing statistics still fall well short of its expectations. It is not surprising that given their inadequacy data on testing are so widely criticised and often mistrusted.

I also welcome the Department’s willingness to work with colleagues from the Office for Statistics Regulation (OSR) and I know they have been in touch to discuss how the data and their presentation could be improved and gaps addressed. OSR will be happy to help further in any way they can.

It would be useful to develop a published timetable for the changes that need to be made and for the development of the metrics for the vital new programme of Test and Trace.

I do understand the pressures that those concerned have faced and still face. But I am sure you would agree that good evidence, trusted by the public, is essential to success in containing the virus.

Yours sincerely,

Sir David Norgrove

Related Links:

Matt Hancock to Sir David Norgrove (12 June 2020)

Sir David Norgrove to Matt Hancock (11 May 2020)

Matt Hancock to Sir David Norgrove ( 27 May 2020)

COVID-19 and the UK Statistical System

Response from Matt Hancock to Sir David Norgrove – COVID-19 testing

Dear Sir David,

Thank you for your letter of 11 May setting out the importance of clear, open and transparent reporting of statistics on COVID-19 tests. I strongly support this, and more broadly the role that high-quality statistics and data play in informing public debate and driving decision-making, and particularly so as we work to address the COVID-19 situation. As you know I have authorised publication of the most full and complete data about our COVID-19 response.

I welcome your positive comments about the information we publish about testing each day. This is a complex and growing programme and our approach to publishing the statistics has developed rapidly. Daily updates are presented on GOV.UK across many different aspects of testing. I now want to go further, and your suggestions for improvement are in line with developments I have already set in train.

We are implementing the following:

  1. We are today publishing a clear definition of how our target of capacity to perform 200,000 tests a day by the end of May will be measured and reported. Capacity will be measured across testing pillars 1, 2, 3 and 4 by asking laboratories each day to set out how many tests they can provide. We will publish today data and commentary on the measure, again on Sunday 31 May, and will repeat, initially weekly, thereafter.
  2. To simplify the presentation of the daily figures on tests and to make the figures easier to find, we are reducing the volume of detailed technical comment each day but will complement this with a detailed note on methods that will also detail any revisions or updates to the figures. This will explain the various strands of testing and will also set out how each strand is counted and reported.
  3. Today we are also publishing a time-series of data, with more detail than our current daily totals, setting out clearly how this has grown over time and making it clear when new strands of testing were added in.

It is not straightforward to provide a definitive geographical breakdown of these figures, but I want to ensure we provide as much detail as possible for those using the statistics and we will continue to develop that analysis. Further, we are reconciling our approach to reporting across the different testing pillars to ensure consistency, and I have asked the Chief Statistician at the Department of Health and Social Care to continue working with you to ensure we provide the best information about COVID-19 testing.

I believe these steps will help to embed the principles of trust, quality and value set out in the Code of Practice for Statistics. I think it is important to demonstrate how good, high-quality data can be used to drive forward our response to this virus, and to build confidence and trust in our figures as we do so.

As you noted in your letter, COVID-19 measurements will need to evolve to ensure they remain meaningful as we move into new phases for tackling the pandemic. As you will be aware the test and trace programme has recently been established under the leadership of Baroness Dido Harding. The programme is committed to being as transparent as possible about its work. Baroness Harding is keen to engage with you on how we ensure the right statistical reporting of the test and trace programme as it develops, and I am copying her into this letter.

Yours ever,

Matt Hancock

Related Links

Sir David Norgrove to Matt Hancock (May 2020)