Executive summary
In November 2022, the nations of the UK signed a statement of agreement covering cooperation on future UK population and social statistics. The agreement requires that an annual report on implementation be prepared for the Inter Administration Committee (IAC). This is the second annual report on the functioning of the agreement, covering the period from October 2023 to October 2024. It includes an assessment of the current UK context and a progress report against the ten areas for cooperation identified within the agreement.
Over the past year, the primary focus of statistical offices across the United Kingdom in implementing the agreement has been cooperation on data-sharing, and in the sharing of technical information about statistical methods. Cross-working mechanisms have developed and adapted to support this. Delivering the agreement requires additional resource across all parts of the UK and the parties have been working together to address this need.
Assessment of current UK context
Since last year’s report, positions on the future of population and social statistics have continued to evolve across the UK. This section summarises the various positions.
The Office for National Statistics (ONS) has published an update (available in English and in Welsh) about its consultation on proposals to transform population and migration statistics. The ONS has continued to develop its admin-based population and migration estimates and publishes quarterly updates. Updated admin-based population estimates for all local authorities in England and Wales, mid-2021 to mid-2022 and provisional estimates for mid-2023 were published in December 2023, and updated estimates for mid-2023 were published in July 2024. An update on research to develop admin-based migration estimates was published in May 2024.
The National Records of Scotland (NRS) will publish a general report on Scotland’s Census 2022 in December 2024. As per legislation, this will be laid in the Scottish Parliament. The report will make initial recommendations for future censuses based on lessons learned. In January 2025, NRS plan to publish an update on research so far into using administrative data for population statistics. NRS has also submitted a business case to the Scottish Government for programme funding.
The Northern Ireland Statistics and Research Agency (NISRA) will publish a general report on their 2021 Census on December 19, 2024. NISRA has bid for funding from the Northern Ireland Executive’s Department of Finance and from the ONS during the year and were partially successful. This new funding has enabled NISRA to begin to build a team to enhance work on data acquisition and analysis, and take forward planning for a user consultation in 2025. NISRA have been developing their own version of admin-based population estimates for Local Government Districts in Northern Ireland and plan to publish a research paper comparing them to the 2021 Census on December 19, 2024. Further research comparing admin-based population estimates to the traditional cohort component method is planned for 2025. NISRA staff have also begun work to identify characteristics available from administrative data sources.
On data-sharing and the use of administrative data, NISRA, NRS and Scottish Government have worked closely with the ONS to improve access to administrative data sources. A priority list of UK administrative datasets was agreed in 2023 and this has been the focus of the work, alongside continued research into using administrative data and building capacity in statistical methods. While some progress has been made, NISRA and NRS have not yet received any additional data. They continue to work with ONS and key data suppliers to address this need, as described in the Progress Report section.
There are different responsibilities across different parts of the UK. Nonetheless, given the interconnected nature of these statistics across the UK, clarity and transparency between nations will be critical. A shared understanding of each nation’s direction will equip all decision makers to best set a direction for population and social statistics that works in the context of each nation, and for the UK as a whole. The UK Statistics Authority’s recommendation for England and Wales is set to be published soon.
Progress report
1. The parties commit to mutual support in the acquisition from third parties of administrative data for statistical purposes.
In line with proposals from last year’s report, the parties have established a Strategic UK Data Sharing Group to improve all parties’ access to data.
The parties have identified a small number of priority data suppliers as important for producing admin-based UK population statistics. The ONS already has access to datasets from most of these suppliers for its research into estimates for England and Wales.
It has been confirmed this year that for all data acquired to date by the ONS, onward-sharing is the preferred short-term route for enabling access to data across the UK; and that the ONS should ensure that user requirements across the UK are clearly defined ahead of all new data negotiations. It has also been agreed that ONS engagement with data suppliers should be, where appropriate, taken forward with a view to ONS enabling an onward data share. This will facilitate the acquisition of datasets of common interest and prevent the need for renegotiations. This approach will also promote coherence through a consistent approach to data engineering and the creation of derived variables.
Onward-sharing is making progress for one supplier, but some complexity has arisen around the selection of specific variables, security and data governance. Lessons have been learned from this experience, and while progress with onward-sharing has been slower than anticipated, the past year’s collaboration and commitments now in place for additional resource mean the parties should expect more rapid progress in onward-sharing over the next year.
The parties have continued to share contacts and guidance for data-sharing requests between themselves. For example, the Scottish Government has shared material and contacts for accessing data from Scottish Government agencies.
Next steps:
2. The parties will continue to work together on cross-border data-sharing required to produce population and internal migration statistics.
The parties continue to meet this commitment through the Internal Migration Working Group.
The ONS aims to use its new admin-based methods for its headline measure of population in mid-year 2025. Over the coming year cross-border data-sharing will continue to be a key priority, but there are not expected to be any changes to data-sharing requirements (in terms of frequency) for population and internal migration estimates in any part of the UK. Further engagement will be needed over time to refine and agree requirements for future estimates.
3. Best practice relating to the efficient use of administrative data sources, including the design of a pipeline to sustain a statistical system reliant on administrative data, will be shared openly.
The parties have a shared list of administrative datasets already used in research, and those they are working to acquire or have future interest in.
A template about the process for the acquisition, engineering, other processing and use of administrative data for onward-sharing has been shared at senior level between UK colleagues. All parties participated in a data pipeline session to compare approaches to processing. This is particularly important given an increased emphasis on onward data-sharing (see section 1).
The ONS is developing a repository of quality metadata that will provide an important resource to understand the strengths and limitations of administrative datasets. It has also developed a data maturity framework that is being used to assess the maturity of data pipelines, which will be shared and developed with the parties.
Next steps:
4. New and developing methods being considered for producing statistics in scope of the agreement should be shared openly and proactively between the parties.
A Task and Finish Group was established in late 2023 to take forward actions in the annual report for 2023. This forum has proven useful and has now been formalised into the FPMS UK Coherence Oversight Group.
To date, technical knowledge-sharing sessions have taken place on the following topics:
- methods to produce a Statistical Population Dataset
- the Dynamic Population Model (with modelling code and preprocessing examples for the DPM shared)
- research into admin-based characteristics estimates
- ONS admin-based population estimates plans for mid-2024 and beyond
Knowledge-sharing sessions on the ONS’s new methods have been positive. Sharing code for these new methods has been particularly valuable. NRS and NISRA have also collaborated on bringing in Bayesian statistical expertise from an international authority to help steer research priorities.
Next steps:
5. Common topics will continue to be used to provide the basis for common outputs, where necessary to meet common user needs.
Since the signing of the agreement, new topics have not yet arisen that require a common approach. This commitment will remain under review as the UK’s statistical system continues to develop. The establishment of UK harmonised topic groups has already demonstrated an excellent way of collaborating on common output interests.
6. Coherent definitions and classifications should be used across the UK.
Differences in user needs, data collection, and legislation across UK nations mean it is not always possible to have completely harmonised data or statistics. The continued development of harmonised standards by the GSS intends to maximise the coherence of outputs. As with survey and census data collection, where administrative sources are to be used for UK statistics the use of harmonised question standards will play a vital role in developing data collection to support coherent statistics.
GSS Harmonisation’s latest workplan, published 1 August 2024, notes three priority areas for development between 2024 and 2026: ethnicity, disability, and sex and gender identity. All other topics are not planned for review until after 2026. The GSS Harmonisation team work closely with colleagues in NISRA, NRS, Scottish Government and Welsh Government.
The GSS Policy and Coordination team, ONS Statistics Head of Profession Office and ONS Coherence teams run Devolution Training sessions for analysts across government, providing guidance on collaborating across the UK when producing outputs relative to other parts of the UK, and promoting the benefits of doing so. All ONS analysts are required to undertake this training. Sessions are run regularly throughout the year and the training has reached over 800 internal staff since the course started, and more colleagues across other government departments through events such Devolution Learning Week, Analysis in Government month, and for the Cabinet Office.
7. Where sources or methods for outputs in scope of the agreement differ, as a priority cross-working mechanisms will be used to maximise the harmonisation of outputs.
UK population estimates for mid-2023, which were published in October 2024, drew on estimates based on the traditional cohort component method across all parts of the UK. For the 2024 mid-year estimates (to be published in 2025), it has been agreed that these should aim to be an aggregate of admin-based population estimates for England and Wales and traditional estimates for Scotland and Northern Ireland, providing any differences between the two approaches are sufficiently understood and agreed by all countries. The management of revisions across different parts of the UK requires further discussion.
The NRS have faced challenges where ONS have needed to adapt plans for National Population Projections to accommodate changes to input data sources in order to continue to meet user requirements. It is recognised that there will be a period of inconsistent methods temporarily while countries are at different stages of the transformation journey.
Methods for producing admin-based statistics on other topics are not yet developed to the extent that they require changes to UK-wide harmonisation efforts. This will be kept under review.
8. The UK Statistics Authority’s ethical principles are to be followed as changes to methods and sources for statistics in the scope of the agreement are considered. The National Statistician’s Data Ethics Advisory Committee will be consulted as appropriate. Findings and recommendations of interest should be shared via cross-working mechanisms.
The ONS recently shared findings and recommendations from engagement with the National Statistician’s Data Ethics Advisory Committee with the devolved statistics agencies and governments. Further findings and recommendations, where of interest, will continue to be shared between the parties in future.
NRS and ONS have collaborated on the ethical basis for transformation work, noting the importance of public acceptability in using linked administrative data. NRS recently presented on its Future of Population Statistics work to the National Statistician’s Data Ethics Advisory Committee and shared findings from this.
9. The parties will share experience of regulatory assessments and checks against the Code of Practice for outputs relevant to the agreement.
In July 2024, following an ONS request for an assessment by the Office for Statistics Regulation (OSR), the OSR published its initial findings and requirements for the ONS to enhance its admin-based population estimates. The ONS has published an action plan in response to the OSR’s assessment and expects to share relevant information through cross-working mechanisms over the next year.
In the running of its census in 2022 a year after the ONS and NISRA, Scotland was able to learn lessons from other parts of the UK in demonstrating compliance with the Code.
10. A common narrative will be agreed and updated at a reasonable frequency to provide users of UK statistics with information about cross-UK cooperation throughout this period of transformation in population and social statistics.
The parties have an agreed joint narrative on the future of UK population and migration statistics, which has been kept under review over the past year. It is set to be refreshed ahead of the publication of the recommendation from the UK Statistics Authority to the UK Government and following key developments thereafter. This should include available information on timelines for coherent UK outputs.
Conclusion
The primary collective focus of the past year has been to ensure that all parties are equipped to investigate administrative-based methods and sources for population estimates, and to a lesser extent to investigate the same for other types of social statistics. Progress with data-sharing has been slower than anticipated, and the proposed next steps within this report, supported by the enhanced cross-working mechanisms that have been developed over the past year and expected additional resource, will be vital to ensuring continued progress during this complex period of transformation.
Related links
Statement of agreement: cooperation on future UK population and social statistics