On Thursday 20 October, Sarah Henry, Director for Methodology and Quality at the Office for National Statistics, gave evidence to the Public Administration and Constitutional Affairs Committee for its inquiry, ‘Civil Service People Survey’.
A transcript of which has been published on the UK Parliament website.
Dear Mr Wragg,
I write in response to the Committee’s call for evidence on the Civil Service People Survey. We have focused our evidence on questions in the Terms of Reference regarding survey design, delivery and validity of results, from the perspective of our role as the Office for National Statistics in administering high-quality national surveys including the census.
Staff surveys, and surveys in general, can adopt different strategies to protect respondents’ privacy. These can range from anonymising responses by removing any information that connects the survey to the respondents, to ensuring that analysis derived from the survey does not lead to disclosing identity.
Full anonymisation can limit analysis. For example, if different age groups have a different experience of working in an organisation, this would not by highlighted if the age field was removed to protect privacy. Therefore, best practice is to seek a compromise by using a range of measures to protect privacy. These include:
In addition, it is good practice to carry out privacy impact assessments and to make privacy notices and technical guides readily available.
To ensure the best design and delivery of a survey, you may want to be aware of the following:
Developing and delivering a survey of the scale of the People Survey is a multidisciplinary task that requires the involvement of many professionals to ensure it delivers on analytical and business objectives.
The information included in the People Survey should be based on the data user needs and the departments that will use it. As mentioned above, these can be ascertained through consultation with policy users. Comparison over time is always an important aspect of any regular survey, and we would recommend keeping question sets as comparable as possible from year to year with changes, when needed, following a transparent methodological review. Finally, some terms used within questions are subjective depending on the department – again this could be improved through consultation.
Periodically the topics covered will change and be impacted by other issues. A good example is the need to monitor the experience of working in the civil service throughout and following the pandemic. When adding or changing a metric, it is important to communicate and explain the changes, especially at the reporting stage.
Some departments may also need to consider organisational changes and how they would like them reported against previous years.
It would be difficult to quality assure the information provided via the People Survey. There are limited sources to cross-check the information, but these could be exit interviews and/or any internal departmental staff surveys. One approach to address this could be to do a quality follow-up survey with a sample of respondents – which is like what we do with the census to quality assure that data.
Non-response bias can have a huge impact. It can cause results to be distorted, and this is linked with wider issues, as typically people that don’t respond have a reason not to engage, and those reasons are particularly interesting for those collecting the data but remain unseen.
It also means that the data will not be representative and, as a result, any policy changes might not address the real issues. Methodological solutions include weighting and imputation and require comparing the population of respondents to the population of civil servants using the data that is available through HR departments. For example, if fewer people aged under 30 respond to the survey, the responses of those who have replied could be given a bigger weight. Any weighting strategy would need to be transparent and carefully considered, attempting to explain the assumption that people who have responded do indeed represent those who have not.
Strengths and weaknesses are mostly the result of trade-offs. For example, while the People Survey is relatively long, risking attrition, lower response rate and haste in completion, it does allow for more detailed analysis on many topics.
As discussed in this submission, using a consistent survey across the Civil Service enables efficiency, comparison between organisations, sharing of good practice and analysis over time while limiting bespoke design on issues that may be of interests to specific departments.
Again, as aforementioned, while the survey is not weighted, which enables quick access to the results, it does have an impact on how confident we can be that respondents represent the civil service as a whole. The survey is reported as percentages of respondents rather than a percentage of the population and users can break down the results further to compare responses from different groups. This is a pragmatic, clear approach which is clearly communicated.
A mixture of both quantitative and qualitative data collection could improve the quality of the analysis and the usefulness of the survey. The People Survey is quantitative, with a few open questions capturing free text. There are other qualitative measures such as depth interviews and focus group discussions that can be used alongside the People Survey to enhance understanding of the results. These can either be in addition to, or instead of some of the questions in the survey.
Finally, the survey is accompanied by a tool that enables quick analysis and comparisons, disseminated to all participating organisations; this is a strength.
My colleague Sarah Henry, Director of Methodology at the ONS, looks forward to discussing this further with the Committee on 13 September. Please do let us know if any questions ahead of then.
Yours sincerely,
Professor Sir Ian Diamond
Dear William,
Thank you for your letter of 14 June 2022 regarding census response rates in Scotland and the implications for the integrity of UK-wide data. I will answer your specific questions in turn, but first I wanted to emphasise the close working relationship between all UK Census offices. We have offered and provided support to National Records of Scotland (NRS) including sharing designs, seconding staff, and are now working with them to develop methods to maximise the accuracy of their Census estimates.
The Office for National Statistics (ONS) also worked with NRS to establish an international steering group, which is providing the highest quality technical expertise, advice and challenge to NRS on census matters. This group is advising NRS to focus efforts toward a good census coverage survey, particularly in regions where responses were lowest, and strengthen the use of administrative data to supplement census data sources in their statistical production, including a clear steer to prioritise the early acquisition of new administrative data sources. This oversight will offer NRS the best possible opportunity to deliver a high-quality outcome for Scotland that will, in turn, contribute to high-quality UK statistics.
The ONS has committed an internationally renowned team of experts equipped with decades of demographic experience to work alongside NRS. Through a combination of this world-leading expertise, the ambitious use of supplementary administrative data, and sophisticated estimation methods, I remain confident that together we will deliver robust UK-wide estimates of the population.
What assessment has been made of the reasons underlying the low response rate in certain areas, and what steps were taken to avoid this occurrence? To what extent is the separate delivery of the Scottish Census considered to have impacted the response rate?
The Census in Scotland is a devolved matter; our assessment of the reasons underlying a low response rate in some areas and the associated mitigations has therefore been formed through our close working partnership with NRS.
The decision by Scottish Ministers to move Scotland’s Census to March 2022 was informed by NRS analysis of the potential impact of COVID-19 on the quality of an operation in March 2021. NRS adopted a diverse and inclusive approach to public awareness around census through media and physical advertising, follow-up reminders, field staff, and focused efforts in areas of low return; approaches comparable to those of the ONS and the Northern Ireland Statistics and Research Agency (NISRA). Evidence gathered in Scotland at the end of the collection phase reported that ‘too busy’ and ‘not aware of the Census or the need to complete it’ were the more common reasons given by householders who had yet to return.
As already mentioned, in light of lower than anticipated returns, the ONS and the Registrar General for Scotland established an international steering group of census and coverage experts. Despite these challenges in the collection phase, having considered the position and the planned next steps with the census in detail, the steering group have confirmed that there is a stable foundation from which to move
from census collection onto the next stage of the census operation, namely the census coverage survey and the incorporation of administrative data into estimates. It is the combination of census returns, coverage survey, administrative data, and estimation methodology that will deliver high quality census outputs for Scotland.
What are the implications of Scotland’s lower response rate for the quality and comparability of UK-wide population statistics?
By taking the actions outlined above, through a combination of census data, supplementary administrative data, and sophisticated estimation methods, we believe that it will be possible to deliver a high-quality outcome for Scotland that will, in turn, contribute to high-quality UK-wide population statistics.
What actions will be taken to quality-assure the Scottish Census data in order to reduce the impact of the lower response rate on the standards of UK-wide population statistics?
NRS remain committed to continuing to produce the best possible population estimates for Scotland, which will be used to produce UK-level population estimates. They currently produce annual official population figures for Scotland using data from a range of sources including the Census, registration data on births and deaths, migration estimates, and a wide range of other administrative data sources. NRS continue to improve these statistics through augmentation of existing and new administrative data sources, ensuring the focus is to deliver population estimates that are accessible and valuable for users.
NRS continue to work closely with partner organisations across the UK, including the ONS and NISRA, to ensure that UK population data and analysis is coherent, comparable and understandable for all users across the UK.
The Committee might also wish to note that the Office for Statistics Regulation (OSR) is currently assessing the Scottish Census, including how NRS are responding to the current situation and the methods and quality assurance they will put in place to provide the best quality data and statistics on the population of Scotland. The OSR has, in both its preliminary findings assessment report for Censuses in the UK, and in subsequent assessment reports for England, Wales and Northern Ireland, highlighted UK data and continues to engage with all three census offices in this regard. I am sure the Director General for Regulation, Ed Humpherson, will keep the Committee informed of its findings, which it plans to publish in November.
Yours sincerely,
Professor Sir Ian Diamond
Dear William,
I am writing to draw your attention to the latest Public Confidence in Official Statistics report (2021), which has been produced by the National Centre for Social Research (NatCen) on behalf of the UK Statistics Authority. I am happy to share that the report finds that public confidence in official statistics remains high, and engagement with official statistics has increased since 2018.
Awareness of the Office for National Statistics (ONS) and the Authority has increased from 70% and 33% in 2018 to 75% and 48% in 2021 respectively. Furthermore, for the first time people were asked if they were aware of the Office for Statistics Regulation, with 41% saying that they were.
Notably, 96% of people able to express a view agreed that it is important for there to be a body such as the Authority to speak out against the misuse of statistics, and 94% agreed about the importance of there being a body to ensure that official statistics are produced without political interference.
Members might also be interested to note that a very high proportion of respondents trusted the ONS (89% of those able to express a view) and our statistics (87%). Of those able to express an opinion, trust in the ONS was highest of all institutions asked about, including the Government, the Bank of England, and the civil service as a whole. 82% of people able to express an opinion agreed that official statistics are generally accurate, up from 78% in 2018. Meanwhile 44% said they had used ONS COVID-19 statistics; they were more commonly used than any of the other statistics asked about with the exception of the census.
This report is very welcome, especially following our hard work to provide clear insights throughout the pandemic. We are proud that the public support our vision of statistics that serve the public good, which we will continue to deliver with honesty, and free from political interference.
A copy of the report will be annexed to this letter for the Committee’s information.
Yours sincerely,
Professor Sir Ian Diamond
On Tuesday 29 March 2022 Sir Robert Chote, the Government’s preferred candidate for Chair of the UK Statistics Authority, gave evidence to the Public Administration and Constitutional Affairs Committee’s pre-appointment hearing for Chair of the UK Statistics Authority.
A transcript of which has been published on the UK Parliament website.
Dear Mr Wragg,
I write in response to the Committee’s call for evidence for the inquiry ‘Coronavirus Act
2020: two years on’; specifically, the transparency surrounding the use of data in decision making relating to the renewal of the Coronavirus Act 2020.
During the pandemic, the response of statistics producers to publish close to real-time data
was remarkable – for example, daily updates on COVID-19 cases and deaths. Producers
demonstrated the ability to consider user needs when balancing timeliness against quality
and strong analytical collaboration resulted in valuable, high-quality statistics.
As noted in our written evidence to your inquiry on data transparency and accountability:
COVID-19, throughout the pandemic the Office for Statistics Regulation (OSR) has called
for greater transparency of data related to COVID-19. In May 2020 we set out our
expectations on the use of management information by government and other official
bodies, and followed up in November 2020 and July 2021.
As you know, unfortunately we had to intervene on multiple occasions regarding instances
of data being quoted publicly that were not in the public domain, which had been used to
inform decision-making. These ranged from data quoted during press briefings, testing data, and data to inform quarantine policy around red-list countries. These were key decisions and announcements made with evidence that the public, media, and Parliament should have been able to scrutinise immediately.
However, two years on, we can see that progress has been made to support the quick
publication of new statistics and data which inform decision-making. It is encouraging that,
when we have raised concerns with producers recently, often during challenging
circumstances such as the emergence of new variants, these have been resolved more
quickly. We discussed the overall improvements in our October 2021 report: ‘improving
health and social care statistics: lessons learned’ where we noted how essential
transparency is for building public trust in statistics and retaining public confidence in
government decisions.
Transparency has been a big theme of our work. In October 2021 we published a blog
setting out our own priorities: build our evidence base of good examples of transparency,
continue to intervene when necessary, and work with external organisation and
governments to make the case for transparency. We are grateful for the Committee’s
support on these issues, and we will continue to keep you updated on this work.
I hope this is a useful summary for the Committee, and please do let me know if I can be of
further assistance.
Yours sincerely
Ed Humpherson
Director General for Regulation
Dear Mr Wragg,
As National Statistician and Chief Executive of the UK Statistics Authority (the Authority), I am
writing to inform the Public Administration and Constitutional Affairs Committee about the UK
Statistics (Amendment etc.) (EU Exit) Regulations 2021, that have today been laid before
Parliament.
The statutory instrument (SI) is a continuation of The UK Statistics (Amendment etc.) (EU
Exit) Regulations 2019 (2019 SI), that came into force on implementation period (IP)
completion day. The 2019 SI revoked 293 pieces of retained EU law on statistics. The aim of
the SI laid before Parliament today is to continue this approach by revoking the remaining
retained EU law on statistics, including new legislation that has come into force since the
2019 SI was laid before Parliament. The SI revokes 137 additional pieces of retained EU law
on statistics.
In revoking retained EU law requiring the onward provision of data to Eurostat, the SI
removes substantive obligations on the UK to collect statistical data at certain times and in
certain ways. While the production of statistics will continue to be subject to domestic
regulation as UK official statistics; the UK will have greater flexibility in the production of these
statistics than under EU law.
HM Paymaster General, the Rt Hon. Michael Ellis QC MP, made the SI in the Cabinet Office’s
capacity as the Authority’s sponsoring department. The Welsh Minister for Finance and Local
Government, the Scottish Deputy First Minister and the Northern Irish Minister of Finance
have confirmed the Welsh and Scottish Government’s and Northern Ireland Executive’s
consent to this instrument respectively.
Please do let me know if any questions or if you wish to discuss further.
Yours sincerely,
Professor Sir Ian Diamond
On 21 October 2021, Sir David Norgrove, Chair, UK Statistics Authority, Professor Sir Ian Diamond, National Statistician and Ed Humpherson, Director General for Regulation, gave evidence to the Public Administration and Constitutional Affairs Committee’s inquiry on the work of the UK Statistics Authority.
A transcript of which has been published on the UK Parliament Website.
Dear Mr Wragg,
I wanted to write to you and your Committee regarding the UK Statistics Authority’s recent establishment of the Centre for Applied Data Ethics, which provides practical support and thought leadership in the application of data ethics by the research and statistical community. This was a direct result from a recommendation from your Committee in 2019, to meet a clear need from stakeholders for leadership in data ethics, both across Government and beyond:
“We recommend that UKSA takes a stronger leading role across technology, data
science, data ethics and influencing improved sharing of data which recognises
UKSA’s role in the governance of the whole system.”
The importance of data ethics cannot be understated as we enable data to be used in ever more radical, ambitious, inclusive and sustainable ways, as set out in our strategy. It is also crucial that the Authority guarantees public trust and acceptability and reduces potential harm to individuals involved in research; both of which can only be ensured through the application of ethical principles.
The Centre empowers researchers to apply ethical principles independently, and we have created an ethics self-assessment tool to that end. This tool has been widely used since its inception, with more than 100 projects this year alone, including those from academia, the Office for National Statistics (ONS), other Government Departments, and Devolved Administrations. We continue to provide expert ethical advice through my independent Data Ethics Advisory Committee (NSDEC), such as for the COVID-19 Infection Survey: which demonstrates how data can be used in an ambitious and timely manner while still being ethical.
The Centre builds on this foundation, providing openly available, practical guidance on a range of cross-cutting ethics topics, as well as developing a new ethics user support service for the research and statistics community and online ethics training for researchers and statisticians across Government, academia and the commercial sector to encourage an ethics-by-design approach to research and statistics. This work is overseen by an independent advisory group consisting of representation from academia and government, who advise on the strategic direction and activities of the Centre.
As part of the Centre’s initial work, we have undertaken a landscape review to situate the work of the Centre within the wider data ethics landscape in the UK. We have also recently released our first piece of guidance on ethical considerations in the use of geospatial data for research and statistics. This guidance was co-created with our user community in response to user needs, and involved engagement and collaboration with a range of professional societies and organisations operating in this area.
As with our plans for all of our guidance, this has initially been released as an early open draft for wider feedback from the user community. It has received positive feedback so far, emphasising the current need for practical guidance, support and resources that can be easily accessed and applied to a range of project types. We also intend to release guidance related to ethical issues in the use of machine learning in research and statistics, considering public good in project applications, and ethical considerations in the use of third-party data later this year, all of which have been developed in response to the needs of, and in partnership with, the user community.
We have a statutory remit to provide support and assurance on data ethics, and our extensive experience means we are regarded as a leader, both in the UK and internationally. We currently lead a workstream on ethics related to the use of machine learning in the production of official statistics on behalf of the United Nations Economic Commission for Europe Machine Learning 2021 group. This involves engaging with collaborating partners from eight countries, as well as with experts in our Data Science Campus, to develop ethical principles and guidance in this area.
I hope this letter is informative about our progress on data ethics in recent years, and of interest to the Committee. The topic is one of growing importance and interest, and I would be happy to discuss further either via correspondence or in person.
Yours sincerely,
Professor Sir Ian Diamond
Dear Mr Wragg,
While giving evidence to your Committee on 18 May, Alex Chisholm, Permanent Secretary at the Cabinet Office, asked that I write on his behalf regarding how we will ensure a coherent and consistent UK wide picture despite Scotland conducting their census in March 2022. I am very happy to provide an update.
Before providing that information, I would like to personally highlight to the Committee that the response to Census 2021 has exceeded all expectations, with returns from 97 per cent of addresses across England and Wales now having been received, surpassing our pre-census target of 94 per cent. The response has been similarly impressive in Northern Ireland with a current return rate of 97 per cent as well. Our next steps include the Census Coverage Survey, which will improve the quality of our final estimates, and our non-compliance operation. We set out in the Census White Paper ‘Help Shape our Future’ that we would lay the initial census results before Parliament within a year of Census Day, and we remain on track to do so.
Moving onto the specific questions around comparability of population data across the UK, there are two aspects to note.
Firstly, as you will no doubt be aware, we produce annual UK population estimates, which are rebased every 10 years based on new information from the latest census. As I previously reported to the Committee in September 2020:
“Certainly when the first indications were made that Scotland was considering the
decision that it has now taken, we brought to our UK census committee a paper that
looked at how we would overcome the problem of re-basing our population estimates
one year apart. I am confident that the strategies that are being proposed to do that
are robust and will not unduly impact on the accuracy of population data over the
next decade.”
The different census dates for the UK will mean that the rebasing process will differ from previous approaches, while ensuring the figures remain comparable. This time, the rebasing process for each country will be staged according to the year in which the census takes place. For England and Wales, the production of the mid-2021 population estimates will be based on 2021 Census data, while for Scotland, they will be rolled forward from mid2020 population estimates. These will feed into the UK population estimate for mid-2021.
We then expect to produce a rebased series of UK population estimates in autumn 2022, covering the period between Census 2011 and Census 2021, which will comprise of census based population estimates for all UK countries bar Scotland, which will again be rolled forward population estimates. In this way, population estimates for the UK for 2021 and 2022 will be based on the latest most accurate information available for each of the constituent countries, whether Census based or rolled forward at the particular time of production, and each using the same definition of usually resident population.
Secondly, there are considerations in respect to the census outputs produced, including UK combined outputs. The census offices continue to work closely together in particular to ensure harmonisation of concepts and definitions wherever possible to support comparability across the UK. These principles are set out in the public agreement between myself and the Registrars General for Scotland and Northern Ireland.
The move to a March 2022 census date in Scotland will mean some changes in the outputs possible compared to those produced following the 2011 Census. We have established a UK Census Data User Working Group to actively engage with UK census users to design the required outputs and explore the possible statistical techniques to provide combined outputs. The work acknowledges the possible rapid change in some characteristics between 2021 and 2022 given the pandemic and other factors. Alongside all outputs we will provide supporting information and guidance to assist users in interpreting combined UK data outputs.
More generally, we will shortly be carrying out a formal consultation on the full range of 2021 Census outputs we plan to produce to ensure that the products made available will best meet the needs of users, especially those users of UK census outputs.
Yours sincerely,
Professor Sir Ian Diamond
