UK Statistical System Code of Practice Office for Statistics Regulation
UK Statistics Authority > Correspondence > UK Statistics Authority

Sir David Norgrove to Miles Briggs MSP – Use of COVID-19 prevalence statistics by Scottish Government

30 July 2020 (by email)

Dear Mr Briggs,

Thank you for your letter of 9 July about a comparison made by the Scottish Government of COVID-19 prevalence in Scotland and England.

The sources used for this comparison were not immediately apparent. As the Office for Statistics Regulation (OSR) has made clear in past statements,[1] when this kind of information is used publicly to inform Parliament, the media and the public, it should be published in an accessible form, with appropriate explanations of context and sources. In this instance, the Scottish Government did not do so.

As you say in your letter, the sources you were provided with do not allow for a meaningful comparison to be made. The Scottish Government have since clarified to OSR that their initial source for prevalence in England was an unpublished estimate by the London School of Hygiene and Tropical Medicine. We do not think that these sources allow for a quantified and uncaveated comparison of the kind being made here.

I enclose a copy of a letter from Ed Humpherson, Director General for Regulation, to Roger Halliday, Scottish Government Chief Statistician, setting out our conclusions in more detail. In future we expect to see clear communication of any data sources used and any limitations or uncertainties appropriately reflected.

Yours sincerely,

Sir David Norgrove

 

Related links:

Ed Humpherson to Roger Halliday: Use of COVID-19 prevalence rates by Scottish Government

Miles Briggs MSP to Sir David Norgrove – Use of COVID-19 prevalence statistics by Scottish Government

Miles Briggs MSP to Sir David Norgrove – Use of COVID-19 prevalence statistics by Scottish Government

Thursday 9th July 2020.

Dear Sir David

I am writing to seek your guidance on the recent use of statistics by the Scottish Government in relation to the prevalence of Covid 19 in the community in Scotland and England.

Scottish Ministers have asserted in recent days that that the number of Coronavirus cases in the community is “five times lower” in Scotland than in England[1]. The Scottish Government has since provided the sources for its claims. The first is the ONS’s publication “Coronavirus (Covid-19) Infection Survey pilot: England, 25 June 2020”[2]. The second is the Scottish Government’s publication “Coronavirus: modelling the epidemic in Scotland (issue No.6)”,[3] also published on 25th June. The Scottish Government’s calculations are based on two figures contained within these publications: the upper threshold for cases: 105,000 in ONS’s survey of England and 2,220 in the Scottish Government’s modelling paper. Applying population share, it then asserts that Scotland’s prevalence rate is 4.7 times lower than England.[4]

I would like the ONS’s assessment of the Scottish Government’s actions in reaching this figure. In particular, two issues appear worthy of attention:

The two figures do not appear to measure the same time period. The Scottish Government’s 2,200 figure is a snapshot estimate for the prevalence of Covid 19 in Scotland on 19th June. The English estimated prevalence rate is the average for the period 8-21 June. The time frames involved are therefore not the same

The methodology used in the two data sets and the purpose of the upper bound is entirely different and therefore not comparable. The ONS figures are based on your Infection Survey of more than 25,000 people in England. The lower-upper threshold on the number of positive Covid 19 cases is therefore a measure of mathematical confidence. The Scottish Government’s figure is based on modelling, mainly from deaths and hospital admissions, and does not therefore carry the same level of certainty as the data in England. The upper threshold for the Scottish data is also is a planning assumption that is not informed by statistical methodologies, so should not be comparable.

It is also notable that the Scottish Government figures used – the two datasets published on June 25th – have now been superseded by new figures which, using this flawed methodology, suggests that the prevalence rate between England and Scotland is in fact broadly similar.

Your attention to these matters would be greatly appreciated.

 

Yours sincerely,

 

Miles Briggs MSP, Lothian.

M2.15 The Scottish Parliament

Edinburgh

EH99 1SP

Related links:

Ed Humpherson to Roger Halliday: Use of COVID-19 prevalence rates by Scottish Government

Sir David Norgrove to Miles Briggs MSP – Use of COVID-19 prevalence statistics by Scottish Government

 

Monica Lennon MSP to Sir David Norgrove: Scottish Government statistics in relation to COVID-19 testing

Dear Sir David,

Following your recent correspondence with the UK Government, I am writing to seek your guidance on Scottish Government statistics in relation to COVID-19 testing.

Proper and transparent use of statistics and data by governments is important at any time but especially during a national crisis. The Scottish Government have regularly stated that they believe that they publish more information and data on the COVID-19 situation that any other UK nation. You may be able to advise if this is the case. My immediate concern, however, is that there are serious issues with the data that is being made available, and the impact this is having on scrutiny.

On that basis, can you advise what assessment the UK Statistical Authority have made of the Scottish Government publications and other public bodies in Scotland?

In your letter to the UK Secretary of State for Health and Social Care you noted both existing issues and suggestions for future publications. I believe a number of these also apply to Scottish data and would welcome your advice on what can be improved in the Scottish context as:

  • The connection between the number of people tested and the number of tests carried out is unclear. Although the publication delineates between the numbers tested through NHS Scotland labs and the UK test centres, there is no information on how many people have been tested through the latter route. Furthermore, it is not clear how many people from either route have been tested more than once, or how many tests results have been void or inconclusive.
  • The information available does not give a full picture of the epidemiology in Scotland. The number of positive cases is likely to be an underestimate as it does not include positive cases identified through UK test centres. The Scottish Government are also carrying out community surveillance programmes but data from these is not regularly published.
  • There is very limited information on the employment of people being tested. Key worker testing is only presented as a total figure, broken down by proportion for types of employment. This makes it more challenging to monitor increases in how many medical staff, care staff, police staff etc. have been tested. There is no detail on how many of these key workers have tested positive, despite regular requests for this information from opposition politicians.
  • Scrutiny of Scottish Government commitments is hampered by a lack of, or incorrect, statistics. More than three weeks after the Scottish Government commitment to regular testing for care homes, statistics on progress were first published on 10 June. However, within hours this had to be reissued due to errors. Information on nosocomial infections has been drip-fed from the Scottish Government who have then claimed that the data released cannot be used for scrutiny because it has not been validated. Finally, the Scottish Government have claimed that there is daily capacity in Scotland for 15,500 tests yet figures on capacity and the proportion used are not made public.
  • On 10 June, data on the Test and Protect system was published for the first time but was insufficient to assess the efficacy of the system. There is no information on the average number of contacts identified per each positive case, no detail on how many contacts have gone on to test positive, and no geographical breakdown. There is also no explanation as to why the number of cases is significantly higher than the corresponding positive test results reported during the same timeframe. Finally, there is no information on how long the contact tracing process takes per case.

Resolving the issues with the UK testing statistics may help address a small number of these issues. Nevertheless, I do consider there to be significant improvements that could be made to data also provided solely by the Scottish Government. I therefore ask that the UK Statistical Authority review these matters and suggest any changes that should be made.

Everyone involved in data collection and publication is working under extraordinary pressure and their efforts are appreciated. The good governance that the public expects relies on the publication of clear, accurate and timely data and robust scrutiny of that data. Public confidence in the Scottish Government’s Test and Protect system will only be achieved through clear and transparent reporting of data.

As you said in your previous letter, good evidence, trusted by the public, is essential to success in containing the virus and I would be grateful for your assessment of what improvements can be made by the Scottish Government and public bodies in Scotland to ensure that information on Covid-19 is transparent and trustworthy.

Yours sincerely,

Monica Lennon

Member of the Scottish Parliament for Central Scotland Region

Scottish Labour Shadow Cabinet Secretary for health and social care

 

Related Links

Sir David Norgrove response to Monica Lennon MSP: Scottish Government statistics in relation to COVID-19 testing

Sir David Norgrove response to Monica Lennon MSP: Scottish Government statistics in relation to COVID-19 testing

Dear Ms Lennon,

Thank you for your letter of 11 June about COVID-19 testing statistics published by the Scottish Government. You point to a number of areas for potential improvement to the published data, and the Office for Statistics Regulation (OSR) have discussed these with Public Health Scotland and the Scottish Government.

In April, OSR reviewed COVID-19 surveillance and registered deaths data across the UK, including information released by the Scottish Government and National Records of Scotland. OSR are in regular contact with producer bodies in Scotland, and a recent blog by our Director General for Regulation acknowledged the wide range of management information being published in Scotland. OSR welcomes the way these bodies are responding to its recommendations, as described for example in a recent Scottish Government blog about future data releases and improvements.

You also raised some particular points in relation to testing data in Scotland.

Testing data are collected through two pathways in Scotland: via NHS Labs (data collected by Health Protection Scotland) and Regional Testing Centres which form part of the expanded UK Government testing programme (data reported through the DHSC from contractors who operate the centres). I understand that data quality issues and timeliness of the data flows from these sites have hindered the release of the data so far by Public Health Scotland. However as of 15 June they are published on the Scottish Government website. I welcome this progress and would encourage further discussion with users and OSR, with a view to continued improvement.

You asked about care home data and Test and Protect figures.

  • On 10 June, Scottish Government published weekly data on COVID-19 in adult care homes in Scotland ‘Coronavirus (COVID-19): adult care homes – additional data’. These data were subsequently revised. I am satisfied that the revision was made and signposted appropriately in accordance with the Code of Practice for Statistics (T3.9).
  • On 10 June, Public Health Scotland published test and protect figures for the first time as part of their weekly COVID-19 report. We understand that the Test and Protect programme is in its infancy and the granularity of the data will be improved in the coming weeks. Scottish Government and Public Health should publish their plans to ensure the public are informed of any developments.

The Office for Statistics Regulation will continue to liaise with Scottish Government and Public Health Scotland as they seek to improve the quality and range of statistics available in Scotland.

Yours sincerely,

Sir David Norgrove

 

Related Links

Monica Lennon MSP to Sir David Norgrove: Scottish Government statistics in relation to COVID-19 testing

Sir David Norgrove response to Stephen Timms MP

Dear Mr Timms,

Thank you for your letter of 2 June about the written response to your Parliamentary Question to ask how many people in 2019 were given leave to remain in the UK subject to the ‘no recourse to public funds’ condition.

We agree that the response was confusing and unclear about why the data could not be provided.

The application of the ‘no recourse to public funds’ condition to those staying in the UK has been a significant focus of attention in past months yet data and statistics to inform the discussion are lacking.

The Office for Statistics Regulation have contacted the Home Office’s Head of Profession for Statistics, who is investigating how good quality statistics can be developed. As the Home Office carries out this work, it should consider user needs and what analyses may be possible, for example breakdowns by age and visa category. In the meantime we would urge the Home Office to look to publish estimates, while recognising that these will have larger margins of error. This would be acceptable under the Code of Practice for Statistics provided the uncertainties were described appropriately.

Yours sincerely,

Sir David Norgrove

 

Related Links

Stephen Timms MP to Sir David Norgrove

Ed Humpherson to Daniel Shaw: Parliamentary question response

Stephen Timms MP to Sir David Norgrove

Dear Sir David,

I have today received the following written answer to a parliamentary question, which I tabled on 20 May:

Question:

To ask the Secretary of State for the Home Department, how many people were given leave to remain in the UK subject to the no recourse to public funds condition in 2019. (49575)

Answer:

Kevin Foster: The information you have requested is not assured to the standard required by ONS for publication and as it would be too costly to do so, we are unable to provide it.

The answer was submitted on 02 Jun 2020 at 13:47.

I found this answer puzzling. What would be involved in assuring the single number I asked for “to the standard required by ONS for publication”? How costly would this be? Do you regard this as an acceptable reason for declining to answer such a straightforward question?

Thank you for your help.

Yours sincerely,

Stephen Timms MP

Chair, Work and Pensions Select Committee

 

Related Links

Sir David Norgrove response to Stephen Timms MP

Ed Humpherson to Daniel Shaw: Parliamentary question response

 

Letter from Sir David Norgrove to Richard Holden MP

Dear Mr Holden,

Thank you for your letter of 4 June about a comparison made on Newsnight between UK daily death figures and the total for the EU.

You raise an important point about the difficulties of international comparisons, which are, as you say, subject to significant differences between countries in definitions and recording practices. We and others have been clear about this[1], and the BBC programme More or Less on 10 June indeed discussed the claim made on Newsnight.

We do not however consider it is appropriate to comment further on this use of statistics in the media. The media every day generate and use immense quantities of statistics, and we cannot take on the task of directly regulating their quality.

More fundamentally, as a public body it is questionable whether we should intervene directly in the media’s use of statistics, save perhaps when the media’s use itself causes a significant risk to public understanding. Parliament, in Ofcom, has specifically created a regulator for the broadcast media. Print journalism is overseen by separate independent bodies including IPSO. In making public interventions on the use of statistics, we are guided by a series of long-standing principles set out in our published policy[2].

I am sorry we are unable to help you further.

Yours sincerely,


Sir David Norgrove

 

  1. [1] For example, questions 36 to 40 of the oral evidence given by the National Statistician and Director General for Regulation to the Public Administration and Constitutional Affairs Committee on 13 May.
  2. [2] statisticsauthority.gov.uk/the-authoritys-role-in-making-public-interventions-on-the-use-of-statistics/

Matt Hancock response to Sir David Norgrove

Dear David,

Thank you for your letter of 2nd June setting out your concerns with our presentation of statistics on testing. We spoke later that day and I understand you have also spoken to Baroness Harding who leads the Test and Trace programme.

The testing programme has been set up at great speed over the last two months in the face of enormous urgent need. While standing up the programme and delivering the substance necessary, we have also sought to produce a range of statistics, in a transparent and open way, to allow people to understand what is being delivered. We are inevitably having to work through some challenging issues as the data systems behind these statistics are developed at great pace. I said in my first letter that I am keen for us to work closely with your team on developing the presentation on the range of information we publish.

Going forward I propose two outputs:

• First NHS Test and Trace will be publishing a weekly release which will set out core metrics on its operational performance. This will focus on the volumes of people covered by the programme, the speed of the user journey through the programme (e.g. test turnaround times) and overall outcome measures. I know that the NHS Test and Trace has been and will continue to engage with your team and the ONS in developing this publication.
• Separately to set out our understanding of the progress of the pandemic through the community my Department is working with ONS, PHE and others to present the best estimates of prevalence, incidence rates and antibodies on an ongoing basis in a co-ordinated way.

ONS and DHSC will work together to ensure these bulletins are coherent and aligned and present a clear overview of the progress of testing and the progress of the pandemic.

While we continue to publish daily updates on testing numbers. I’ve asked my officials to continue to work with you on improving the presentation of these. I have asked the Department’s Head of Profession for Statistics – Stephen Balchin – to write to the Office for Statistics Regulation with periodic updates on the specific points in your letter.

Yours ever,
Matt Hancock

 

Related Links

David Norgrove to Matt Hancock (June 2020)

Sir David Norgrove response to Matt Hancock regarding the Government’s COVID-19 testing data

Dear Secretary of State,

Thank you for your letter of 27 May, in which you described some welcome, though limited, additions to the official data on COVID-19 tests, including a proposed note on methods (not yet published at the time of writing). I am afraid though that the figures are still far from complete and comprehensible.

Statistics on testing perhaps serve two main purposes.

The first is to help us understand the epidemic, alongside the ONS survey, showing us how many people are infected, or not, and their relevant characteristics.

The second purpose is to help manage the test programme, to ensure there are enough tests, that they are carried out or sent where they are needed and that they are being used as effectively as possible. The data should tell the public how effectively the testing programme is being managed.

The way the data are analysed and presented currently gives them limited value for the first purpose. The aim seems to be to show the largest possible number of tests, even at the expense of understanding. It is also hard to believe the statistics work to support the testing programme itself. The statistics and analysis serve neither purpose well.

To mention just a few issues in relation to the data as currently presented:

  • the headline total of tests adds together tests carried out with tests posted out. This distinction is too often elided during the presentation at the daily press conference, where the relevant figure may misleadingly be described simply as the number of tests carried out. There are no data on how many of the tests posted out are in fact then successfully completed. The slides used in the daily press conference do not show the date when the tests were carried out;
  • the notes to the daily slides rightly say that some people may be tested more than once and it has been widely reported that swabs carried out simultaneously on a single patient are counted as multiple tests. But it is not clear from the published data how often that is the case. Figures for the overall number of people being tested have previously been published but are not available in the published time series;
  • the top summary presents the number of positive results from diagnostic tests (pillars 1 and 2) alongside the total number of tests across all pillars. This presentation gives an artificially low impression of the proportion of tests returning a positive diagnosis;
  • more generally the testing figures are presented in a way that is difficult to understand. Many of the key numbers make little sense without recourse to the technical notes which are themselves sometimes hard to follow. This includes the supporting spreadsheets, which, while welcome, make it difficult to extract even basic trends.

With regard to new data that are not currently made available:

  • test results should include for example key types of employment (e.g. medical staff, care staff), age, sex and location (by geography and place, such as care homes). How many people in what circumstances are infected? Where do they live?
  • for Test and Trace it is important that a statement of the key metrics to measure its success should be developed systematically, and published, to avoid the situation that has arisen in relation to the testing programme. The statistics will need to be capable of being related to the wider testing data and readily understood by the public, through for example population adjusted maps of hotspots.

I warmly welcome of course your support for the Code of Practice for Statistics. But the testing statistics still fall well short of its expectations. It is not surprising that given their inadequacy data on testing are so widely criticised and often mistrusted.

I also welcome the Department’s willingness to work with colleagues from the Office for Statistics Regulation (OSR) and I know they have been in touch to discuss how the data and their presentation could be improved and gaps addressed. OSR will be happy to help further in any way they can.

It would be useful to develop a published timetable for the changes that need to be made and for the development of the metrics for the vital new programme of Test and Trace.

I do understand the pressures that those concerned have faced and still face. But I am sure you would agree that good evidence, trusted by the public, is essential to success in containing the virus.

Yours sincerely,

Sir David Norgrove

Related Links:

Matt Hancock to Sir David Norgrove (12 June 2020)

Sir David Norgrove to Matt Hancock (11 May 2020)

Matt Hancock to Sir David Norgrove ( 27 May 2020)

COVID-19 and the UK Statistical System

Response from Matt Hancock to Sir David Norgrove – COVID-19 testing

Dear Sir David,

Thank you for your letter of 11 May setting out the importance of clear, open and transparent reporting of statistics on COVID-19 tests. I strongly support this, and more broadly the role that high-quality statistics and data play in informing public debate and driving decision-making, and particularly so as we work to address the COVID-19 situation. As you know I have authorised publication of the most full and complete data about our COVID-19 response.

I welcome your positive comments about the information we publish about testing each day. This is a complex and growing programme and our approach to publishing the statistics has developed rapidly. Daily updates are presented on GOV.UK across many different aspects of testing. I now want to go further, and your suggestions for improvement are in line with developments I have already set in train.

We are implementing the following:

  1. We are today publishing a clear definition of how our target of capacity to perform 200,000 tests a day by the end of May will be measured and reported. Capacity will be measured across testing pillars 1, 2, 3 and 4 by asking laboratories each day to set out how many tests they can provide. We will publish today data and commentary on the measure, again on Sunday 31 May, and will repeat, initially weekly, thereafter.
  2. To simplify the presentation of the daily figures on tests and to make the figures easier to find, we are reducing the volume of detailed technical comment each day but will complement this with a detailed note on methods that will also detail any revisions or updates to the figures. This will explain the various strands of testing and will also set out how each strand is counted and reported.
  3. Today we are also publishing a time-series of data, with more detail than our current daily totals, setting out clearly how this has grown over time and making it clear when new strands of testing were added in.

It is not straightforward to provide a definitive geographical breakdown of these figures, but I want to ensure we provide as much detail as possible for those using the statistics and we will continue to develop that analysis. Further, we are reconciling our approach to reporting across the different testing pillars to ensure consistency, and I have asked the Chief Statistician at the Department of Health and Social Care to continue working with you to ensure we provide the best information about COVID-19 testing.

I believe these steps will help to embed the principles of trust, quality and value set out in the Code of Practice for Statistics. I think it is important to demonstrate how good, high-quality data can be used to drive forward our response to this virus, and to build confidence and trust in our figures as we do so.

As you noted in your letter, COVID-19 measurements will need to evolve to ensure they remain meaningful as we move into new phases for tackling the pandemic. As you will be aware the test and trace programme has recently been established under the leadership of Baroness Dido Harding. The programme is committed to being as transparent as possible about its work. Baroness Harding is keen to engage with you on how we ensure the right statistical reporting of the test and trace programme as it develops, and I am copying her into this letter.

Yours ever,

Matt Hancock

Related Links

Sir David Norgrove to Matt Hancock (May 2020)