Dear Mr Hoare,
Now that the Committee has been reconstituted, I write to provide a response from the UK Statistics Authority to your predecessors’ report on ‘Transforming the UK’s Evidence Base’, published shortly before the General Election was called. On behalf of the Authority, I would like to express my thanks to them and their supporting staff for launching this timely inquiry and for the report and recommendations.
This report comes at an important time for the official statistical system, with the Authority due to convene the first ‘Statistical Assembly’ and prepare its next strategy for the statistical system, due to be published in July 2025. In the coming months the Authority will also be making a recommendation to government on the future of population and migration statistics in England and Wales, based on advice from the National Statistician.
In sending this response, I would also like to highlight to the Committee our response to Professor Denise Lievesley’s Independent Review of the Authority, which raised many similar issues and themes as the Committee’s report. Below, I broadly address the Authority view of the key points from each section of the Committee’s report and respond to recommendation 5 in more detail. Appended to this, you will also find the individual responses of Professor Sir Ian Diamond, National Statistician and Ed Humpherson, Director General for Regulation on behalf of the Office for National Statistics (ONS) and Office for Statistics Regulation (OSR) respectively, addressing their specific recommendations.
The Authority welcomes the Committee’s engagement on the future of the UK’s statistical system and the opportunities presented by your recommendations. We will continue to keep you updated on our work and progress made towards the recommendations aimed at the Authority, the ONS and OSR respectively.
Yours sincerely,
Sir Robert Chote
Chair, UK Statistics Authority
Delivering evidence for the public good
Recommendation 5: It is time to democratise access to data and evidence. The UK Statistics Authority should establish a framework for identifying and prioritising demands for evidence. We recommend that it use a high-level Assembly (of the kind recently recommended by Professor Denise Lievesley) to draw together information from communities across the UK about their needs for evidence and the benefits new evidence would bring, alongside research on data gaps, and public understanding. We further recommend that the UK Statistics Authority submit its findings on the nation’s demands for evidence to Parliament on a triennial basis, for scrutiny by this Committee.
1. The Authority welcomes the Committee’s fifth recommendation, that both a framework and a high-level assembly be established to identify and prioritise demands for evidence, with its findings submitted to PACAC for scrutiny on a triennial basis. Work is already underway to meet these objectives. In April, the ONS published its Strategic Business Plan for 2024-25, setting out our approach to prioritisation in a constrained financial environment. The Plan makes a commitment to securing the stability and quality of our core statistical outputs across five priority areas. In taking on additional work, the ONS will seek to align its resources to activities and programmes where it is uniquely placed to deliver, and that have the highest impact on the strategic priorities.
2. As set out in the Authority’s response to the Lievesley Review, a UK Statistics Assembly will meet for the first time on 22 January 2025. It will bring together users and producers across sectors to discuss and give advice on the UK’s needs for statistics. The insights will be drawn together in a published advisory report, indicating potential priorities and data gaps for scrutiny by users and by your Committee. They will inform delivery planning for the ONS and other official statistical producers.
3. As well as identifying data gaps, the Assembly’s discussions will provide valuable insights on the quality of statistics, to contribute to shaping the OSR’s regulatory work programme. Following the first Assembly in January 2025, the Authority and stakeholders will review the frequency of future events, including the timing of future reporting to Parliament. As Professor Lievesley has pointed out, there is no precise template for an Assembly so the first will inevitably be an experiment from which we can learn.
4. Additionally, across the ONS, producers of statistics regularly engage with users of statistics across existing forums and advisory groups. For example, the ONS Local team (based physically in the nine regions of England as well as Scotland, Northern Ireland and Wales) act as the front door for local government to access the ONS and the wider GSS, supporting users to make the most of a wide range of data and analysis.
5. Furthermore, the ONS provides accessible digital content to help audiences find, understand, explore, and act upon its content. These include data visualisations and explorer tools, as well as explanatory articles and bite-sized videos to suit different audiences – available both on its own website and via external channels, including social media platforms. To extend the reach of statistics and data to audiences with whom the ONS traditionally has had less engagement, the ONS works with relevant organisations and citizen representative groups, to help disseminate its outputs as well as inform the design of communications.
6. The ONS have also initiated a Citizen Data project with the aim of securely playing back data held on the citizen to the citizen. This will enable ONS to engage directly on a one-to-one basis encouraging individuals to validate their personal data and help to improve characteristics coverage and public trust in the use and storage of administrative data.
7. OSR continuously seeks to embed the principle of statistics for the public good in its regulatory approach. The Code of Practice for Statistics has clear expectations that official statistics support the needs of a wide range of users, alongside policy makers. OSR is in the process of refreshing the Code and will continue to articulate and strengthen its expectations on this principle. One of the areas of focus for the refresh of the Code is a greater focus on user engagement.
8. OSR also conducts research to further understand how official statistics can serve the public good. In its recent think piece, OSR proposed that “official statistics serve the public good as public assets that provide insight, which allows them to be used widely for informing understanding and shaping action”. OSR is also undertaking complementary research into how individuals may use statistics in their personal lives. This research is used to strengthen its regulatory approach, and its advice and requirements on producers of official statistics.
9. As mentioned, the Authority’s current strategy, Statistics for the Public Good, launched in 2020 and will end in July 2025. We will look to engage with the Committee to ensure the next strategy reflects views from users, including Parliament.
Navigating new data sources and technologies
10. The Authority acknowledges the Committee’s high-level findings that there has been an increase in data generated across the UK, with a need to bring together ‘old’ and ‘new’ data sources to make best use of it. Valid concerns are raised about the current provision and proficiency of cross-government data sharing and how if not addressed, this may hamper efforts such as our ambitious future of population and migration statistics programme, which is seeking fully to capitalise on the transformational opportunities offered by administrative data sources.
11. As we outlined in our response to Professor Lievesley’s review, we concur with the notion that a drive is needed from the centre of government to increase incentivisation and ability for departments to share data between them, with the FPMS programme and successful delivery of the Integrated Data Service (IDS) requiring a positive culture shift towards data sharing becoming a reality. Additionally, we seek to be as transparent as possible about what data we are seeking and how we are using them. Thus, we see the benefit in all the recommendations in this section of the report and they are explored in more detail in the ONS response.
Evidence in policymaking
12. Several recommendations in this section are made with a view to ensuring the Analysis Function (AF) has the resource and vision it requires to enact significant change and evaluate its future successes. As part of the privacy section of the report, the Committee also suggests that the AF explore options for improving transparency where personal data is used in official analyses. These recommendations are responded to at length in the ONS’s response, given the National Statistician’s role as Head of the AF.
13. I was pleased to note the Committee’s praise for the OSR’s fantastic work on Intelligent Transparency (IT) and their suggestion that its remit should be widened in scope and government communication professionals be trained in the IT principles. Previous work that the OSR has done in this space, and thoughts on these recommendations, can be seen in their response.
Privacy and ethics in an age of data
14. The Committee’s last recommendation that the Centre for Data Ethics and Innovation should continue to monitor public attitudes on the Government’s use of data is welcomed by the Authority.
15. The Authority pays tribute to the work carried out by the Responsible Technology Adoption Unit (RTAU) in the Department for Science, Industry and Technology (previously CDEI). We also monitor public attitudes towards the use of data and trust in institutions more widely, including an insights paper we published in June 2023. This release summarises our findings on public attitudes, concerns and expectations on the use of data and views on our use of administrative data in publishing statistics.
16. In the past, the Authority has regularly commissioned the National Centre for Social Research (NatCen) to assess independently the public’s knowledge of, and trust in official statistics, and how they are produced and used in the Public Confidence in Official Statistics (PCOS) Survey. The most recent PCOS survey results were published by NatCen on 14 May 2024.
17. Additionally, in 2022 and 2023 the ONS was commissioned by the Cabinet Office to run the Organisation for Economic Co-operation and Development Survey on Drivers of Trust in Public Institutions on behalf of the UK Government. The most recent release, for 2023 was published on 1 March 2024. We therefore support this recommendation and would be happy to provide our knowledge and expertise to RTAU to assist with future work seeking to understand public attitudes towards data usage.
Sir Robert Chote, Chair
UK Statistics Authority
The Office for National Statistics (ONS) response addresses the Committee’s recommendations both directly aimed at the ONS, and those where other government departments have joint responsibility. This response focuses on data sharing and the future of population and migration statistics programme, which are inherently linked, and data ethics. It also provides a response to the series of recommendations specifically directed at the Analysis Function.
Data Sharing and the Future of Population and Migration Statistics
Recommendation 1: It is time for Government to do what it promised to do seven years ago, and to join up the UK’s evidence base. Given that the Cabinet Office’s existing initiatives for improving data sharing are self-evidently insufficient, it should in partnership with the Office for National Statistics develop a comprehensive new programme aimed at improving data-sharing for statistical and research purposes. The programme must clearly define deliverables and timelines, and must be owned by a senior responsible officer at an appropriately high level. In line with the recommendations of the Lievesley report, we also recommend that HM Treasury establish mechanisms so that the costs are not borne by individual Departments, but rather centrally. The Cabinet Office should prepare and publish an annual progress report on delivery against the programme.
18. The ONS is strongly supportive of efforts to enhance data sharing across Government. As the largest producer of official statistics, we are dependent on effective data sharing across the public sector and beyond, to support more quality, timely and granular admin-based statistics. The ONS also plays a key role in supporting Government, the devolved administrations and wider academia to access data to support statistical research. As such, the ONS firmly supports the Committee’s recommendation that a cross-government data sharing programme be established.
19. To date, we have worked closely with the Central Digital and Data Office (CDDO) and wider government departments to promote effective data sharing. We also played a leading role in supporting key initiatives to deliver upon the commitments within the 2022-2025 Roadmap for Digital and Data. We have supported a number of initiatives aimed at improving data sharing, including developing the Data Maturity Assessment for Government; the identification (and publication) of Essential Shared Data Assets and developing common governance arrangements to support sharing of data.
20. As the lead delivery partner for the Integrated Data Service, the ONS has also delivered a trusted research environment in the cloud. We are uniquely well placed to facilitate access to a growing library of linked data sources to support collaborative analysis, including to support the development and delivery of Government’s key missions. However, this will only be possible with continued and increased support from key data owners across government.
21. The ONS recognises both the progress that has been made and the substantive challenges that remain to cross-Government data sharing. Therefore, the ONS welcomes the creation of a new Digital Centre of Excellence within the Department for Science, Innovation and Technology. We look forward to working with DSIT to define and implement a programme of work that drives a step change in data sharing to enable statistical and research use cases within Government and beyond.
Recommendation 2: Separately, the Office for National Statistics should publish information on the datasets it is seeking on an annual basis, setting out its rationale for seeking those data, and details on the status of the request – all of which should be made available on the ONS website.
22. The ONS accepts the Committee’s recommendation that we publish information on, and rationale for, the datasets we are seeking on our website annually. This fits well with our strong desire to be transparent about the data we use to support statistical outputs. For example, we already publish a report on the datasets that we have acquired that contain personal identifiers which was most recently updated in July 2024. We are working to expand the coverage of this publication to cover a broader array of alternative and administrative data sources, irrespective of whether the dataset contains personal identifiers or not.
23. We have a broader transparency ambition, which will lead to further publications that provide information about how the data we acquire are processed and the relationship between these data ‘inputs’ and our broad portfolio of statistical outputs. Whilst all the necessary information is available on an individual basis, drawing together all of the elements needed to depict this will require a programme of work across the office, including the development of an enterprise data model. We intend to start with some illustrative examples to test the best ways of presenting what will be a very large amount of information. We will then expand from that point, recognising the need to be both informative and comprehensive.
24. We also acknowledge the importance of being transparent about the data that we have not yet acquired, in both illuminating the progress on key data shares, but also conveying a clear sense of our progress in delivering a viable administrative data based statistical system. Therefore, we agree to add data that is in the process of being acquired to our transparency reporting. We will
develop the best format for these publications, in conjunction with our various suppliers so that we can appropriately convey the status of an acquisition.
25. We recognise that data sharing is a complex process. Various stages are required to mature sharing arrangements and deliver sustainable supplies of data and it is necessary to provide a sense of how mature our sharing arrangements are. We must also ensure that we adhere to commercial sensitivities in naming some suppliers.
Recommendation 4: This Committee’s view – particularly in light of challenges around data-access – is that officials have not yet demonstrated that they can deliver the evidence users need, without a decennial census. We therefore recommend that the Office for National Statistics undertake further work on proposals for the future of migration and population statistics.
26. Both the ONS and the Authority welcome the Committee’s recognition of the opportunities offered by administrative data sources. We also recognise the need to improve the culture of data-sharing across government if we are to maximise those opportunities. This is a challenge that was also highlighted by Professor Denise Lievesley’s review earlier this year, and which the Authority continues to work with partners across the public sector to resolve.
27. Data already held within the public sector mean population and migration statistics can be more consistently accurate and produced more often and quickly. As a result, decision-makers have more, higher-quality, information about local populations, their characteristics, where they live and the public services they need.
28. In line with the Committee’s recommendation, the ONS continues its work to develop and improve admin-based population estimates, using innovative new methods and a wider range of data sources, accounting for quality limitations in the data. We published updated estimates as official statistics in development in July, and aim for these to become the official mid-year population estimates in 2025.
29. The Authority expects to publish its recommendation on the future of population and migration statistics in England and Wales in the coming months. This recommendation will draw on extensive engagement with users of these statistics, including through the public consultation last year, and will include the Authority’s proposed approach to the future of the census in England and Wales.
The Analysis Function
Recommendation 10: We recommend that Government reaffirm its commitment to the analysis function, and that HM Treasury review options for its future funding. If Government truly wishes to improve its use of analysis and deliver better outcomes for the public, it clearly needs to fund that change.
30. The ONS remains committed to the Analysis Function (AF), and as such accepts the principle of this recommendation and is grateful for the Committee’s concern about its future funding model.
31. We believe that funding for a dedicated central AF team is essential to ensure that analysts across the Civil Service have the support they need to deliver better outcomes for the public by providing the best analysis to inform decision making.
32. Therefore, the AF Central Team (AFCT) will work with HM Treasury to assess the best option for future funding. As Head of the AF, I will work with Chief Analysts across government to ensure the profile of the Function continues to be raised.
Recommendation 11: In parallel, the National Statistician should review the analysis function’s scope and standard, with a view to defining an achievable set of next-steps, and clear plans for honest evaluations of the function’s success. This review and subsequent evaluations should be made publicly available, so that Parliament is in future better equipped to scrutinise both the Government’s use of evidence and the progress of the analysis function.
33. The ONS accepts the Committee’s recommendation to review the AF’s scope and standard with achievable next steps in mind and will take the points raised away for further consideration. Subject to gaining sufficient funding for the AFCT, the team will review the scope of the AF, which will be reflected in the updated AF Strategy for 2025-2028. The team will also review the AF standard, focusing on any changes needed to reflect the importance of transparency in analysis.
34. With regards to the evaluation of the AF, subject to funding, the AFCT will evaluate the impact of their work to support analysts across government. It will also undertake a light touch assessment of the impact of analysis more widely, using existing evidence sources, such as the results of the assessment against the AF Standard. This will be complemented by the work of departmental Chief Analysts who are responsible for evaluating the impact of their analysis and whether it is meeting the needs of their decision makers.
35. The AFCT will update the Committee on the findings of this evaluation, via a letter from the National Statistician to the Chair. The AFCT anticipate that this work will be completed by Q2 2025/26. However, further reviews of the standard, and evaluation of the work of the Central Team, will be undertaken as part of our business as usual.
Recommendation 14: We recommend that, at a minimum, governments in future routinely publish the evidence and data underpinning their major policy announcements. Making this happen will not be a straightforward task, and we suggest that in the first instance leaders of the analysis and communications functions develop options to deliver this ambition, for the consideration of Ministers.
36. The ONS accepts the Committee’s recommendation that options be developed for Ministers on the routine publication of evidence and data underpinning major policy announcements in the future. Transparency in evidence and data underpinning policy decisions is an important matter for the AF and was discussed at an AF Board meeting last year. There is a variety of guidance already in existence in this area, including the Code of Practice for Official Statistics, the Analysis Function Standard and OSR’s guidance on Intelligent Transparency.
37. In line with the Committee’s suggestion, the AFCT will work with the Communications Function, and other bodies, such as the Policy Profession, to consider options around the recommendation to routinely publish the evidence and data underpinning their major policy announcements.
Recommendation 15: We recommend that the analysis function explore options for improving transparency around the use of personal data in official analyses, and that this work be made publicly available.
38. The ONS accepts the recommendation that the AF explore options for improving transparency around the use of personal data in official analyses. Subject to sufficient funding, the AFCT will investigate options around what would improve transparency around the use of personal data in official analysis, working with relevant bodies that deal with use of personal data such as CDDO and OSR. The AFCT will complete this work by Q4 2025/26.
Data Ethics
Recommendation 16: It is now time to consolidate the excellent exploratory work that has been done on data ethics, and to embed it more formally into the collection, analysis, and communication of evidence in the UK. We recommend that the Cabinet Office’s Central Digital and Data Office and the Office for National Statistics jointly review the varying data ethics frameworks available to analysts across the UK; considering opportunities for greater consistency, and possible accountability mechanisms, to encourage a wider adoption of data ethics across government.
39. The Authority has worked with the CDDO to develop a common understanding of data ethics in the public sector. Our conversations have resulted in agreement with the recommendation that existing frameworks across the UK be reviewed with the aim of encouraging wider adoption of data ethics across government.
40. Regular working-level meetings between the data ethics teams from both departments have been organised, and discussions have included sharing each team’s learnings in the data ethics space. Together, we have discussed the Authority’s data ethics self-assessment tool and the recent landscape review around the responsible use of data-driven technologies in the public sector for which the Authority participated in an interview, amongst others. The Authority’s Centre for Applied Data Ethics (CADE) continues to monitor the impact of the tool along with providing practical support and thought leadership in the application of data ethics by the research and statistical community.
41. As noted in the Government response, recommendations coming out of the landscape review suggested consolidation work on data and AI ethics guidance across government. We will work with CDDO and other partners across government on this exercise.
42. We also wish to concur with the point raised in the Government response about flexibility and context. We agree that harmonisation is desirable in some instances and have discussed shared opportunities with CDDO. But our objective is to promote and safeguard the production and publication of official statistics, and specifically provide guidance to researchers (from within and outside of government) on the ethics of their research. Therefore, we agree that much of the guidance material we produce must be discrete from general data ethics guidance produced for central government.
Professor Sir Ian Diamond, National Statistician
Office for National Statistics
September 2024
This response from the Office for Statistics Regulation (OSR) addresses the Committee’s recommendations for OSR as well the recommendations with joint responsibility across other government departments. Our response has focused on the development of a framework for reporting data gaps across the UK, and on provisions for improving intelligent transparency across government.
Data gap reporting framework
Recommendation 6: We recommend that the OSR support this activity by preparing regular and public reports on data gaps in the UK.
43. OSR accepts the Committee’s recommendation to prepare regular and public reports on data gaps in the UK in principle.
44. Identifying and helping statisticians focus on addressing data gaps is an important aspect of our regulatory work. Our domain (topic area) model of regulation allows us to have strong relationships with statistics producers and organisations who publicly advocate for improved statistics. Our annual report, State of the Statistical System, which was last published on 17 July 2024, is our key publication bringing together the insight provided by our regulatory work. It also shares our views on the performance of the statistical system and the challenges facing it. In line with the Committee’s recommendation, we will have an enhanced focus on data gaps in future editions of this report.
45. Our recently published report, “Data Sharing and Linkage for the Public Good: Follow-Up Report” highlights the importance of enabling greater data sharing and linkage, in a secure way, for research and statistics22. This is also relevant to the government’s ability to respond to data gaps.
46. Additionally, OSR is refreshing the Code of Practice for Statistics to ensure that it remains relevant. We are strengthening the emphasis on involving users in decisions about what statistics are required – whether to start, stop or change official statistics. This includes being clear on where and why user needs can and cannot be met, such as addressing information gaps.
47. Noting recommendation 7, in our aforementioned “Data Sharing and Linkage for the Public Good” report, we call for consistent and sustainable funding streams for data sharing, access and linkage initiatives, and specifically for a centralised government funding structure to support data collaboration projects across government. This structure should prioritise a system-level, access-based approach to investment, as well as continue and expand initiatives such as the Shared Outcomes Fund.
Recommendation 9: We recommend that the Office for Statistics Regulation review and publish a report on the adequacy of UK-wide comparable data, by themes, before April 2025.
48. OSR accepts the Committee’s recommendation that we review and publish a report on the adequacy of UK-wide comparable data. Our State of the Statistical System report highlighted the need for producers to work in partnership across the UK and provided examples of where statistics producers are making improvements to UK comparable statistics and data. We will build on this work and publish an update in 2025 which shares our more detailed views on the adequacy of UK comparability.
Improving intelligent transparency across government
Recommendation 12: We commend the OSR for its work on Intelligent Transparency and recommend that it publish an annual report card on departments’ compliance with its guidance, so that Parliament and external bodies might support it in holding departments to account, and making the case for well-informed policy. Recognising that this important work expands the remit of the OSR beyond official statistics, and into the larger world of government analysis, we also suggest that at the next Spending Review, it works with HM Treasury to agree a sustainable funding model for this work, given the vital role it plays.
49. OSR accepts this recommendation and thanks the Committee for its commendation of our work on intelligent transparency (IT). We agree that this would expand the remit of OSR and would therefore need to be appropriately funded.
50. OSR has an intelligent transparency working group tasked to find ways to promote, and embed, the IT principles across government. We are beginning work to develop proposals for a monitoring and reporting approach for IT across departments. The project will consider incentives and formats for departmental reporting, as well as automated tools for OSR to monitor IT on a rolling basis. We will consider these proposals alongside discussions with HM Treasury on a sustainable funding model for this work.
Recommendation 13: We recommend that all government communications professionals are trained on the OSR’s Intelligent Transparency guidance, and that the Government Functional Standard for Communication be updated to make it clear that officials are expected to comply with that guidance.
51. OSR accepts the recommendation that government communications professionals should receive training on IT and would be happy to work in partnership with the Cabinet Office to achieve this.
52. Since the creation of the IT principles, OSR has continued to promote them across government, working with a range of organisations and professions, including the communications profession. We are very encouraged by the response and engagement from departments to date, and the commitment to supporting public confidence in government statistics and data through transparency. Our ambition is to continue to take these principles to new audiences, including ministerial private offices and Special Advisors.
53. We are also exploring ways to provide our IT materials on our website following feedback from sessions we have delivered previously. This includes developing new guidance on adhering to IT principles on social media.
54. As part of our work refreshing the Code, we are looking to articulate our standards relating to IT more clearly and better highlight how they relate to all those in government involved in communicating statistics, data and wider analysis.
55. Relatedly, we wish to express our support for recommendation 14 of the Committee’s report, that future governments should publish the evidence and data underpinning their major policy announcements. We already encourage IT around policy announcements and government decision making and will continue to do so. Our latest report on Analytical Leadership explores our position on this in more detail. Additionally, I included this issue in my letter to Permanent Secretaries at the start of this year’s general election campaign.
Ed Humpherson, Director General for Regulation
Office for Statistics Regulation
September 2024