Dear Mr Hoare,
Following evidence I gave to your Committee in July, and a subsequent request for supplementary evidence, please find additional information on the Transformed Labour Force Survey (TLFS), Planning & Portfolio management, and the Integrated Data Service (IDS) enclosed.
Please do let me know if I can be of any further assistance.
Yours sincerely,
Emma Rourke
TLFS
Can you please provide the initial project documentation for TLFS?
The TLFS was created from a series of legacy research and delivery projects over many years. The complexity of this evolution underpins some of the challenges the project has faced. The timeline of the TLFS is enclosed at Annex A. The first prototype of this survey was called the Labour Market Survey (LMS), which was later renamed the TLFS.
The high-level design for the TLFS was originally based on design concepts developed within the Data Collection Transformation Programme, which were presented to and endorsed by the Office for National Statistics (ONS) National Statistics Executive Group (NSEG) in May 2017 as part of the solution to falling Labour Force Survey (LFS) response rates. This paper and its annexes are enclosed at Annex B.
This design was taken forward for development within the Integrated Population and Characteristics Survey (IPACS) project in June 2019 under the Census and Data Collection Transformation Programme (CDCTP). A paper on the development of the IPACS was presented to the Methodological Assurance Review Panel in June 2019 and is enclosed at Annex C.
The fundamental features of the IPACS design were:
- An online first survey.
- A respondent centred design approach which engaged respondents in its creation and minimised respondent burden.
- An integrated design – which brought together the existing multiple disparate surveys across the ONS into a single master survey with accompanying topic-based follow-up modules (i.e. labour market, household expenditure etc). See Figure 1 in Annex C.
- The significant use of administrative data: to inform both survey sampling, and to enable surveys to be significantly shortened through the replacement of questions on data already held by government, such as house valuation, council tax, income etc. See Figure 1 in Annex C.
In developing and implementing this original survey design, the TLFS lessons learnt review, published in December 2024, outlines key issues that were encountered: “The original IPACS design was for a modular system with a combination of surveys and administrative data to meet the wide range of user requirements. With the lack of available administrative data, organisational priorities in delivering Census and the impact of the pandemic, TLFS subsequently became the de facto solution for meeting all user needs rather than just the core labour market.”
This led to the loss of the integrated design approach and the loss of administrative data to shorten the survey. Without administrative data to replace survey questions, the LMS was longer than originally intended in the concept but still aimed to be substantially shorter than the LFS (200 to 300 variables on the LMS vs 604 variables on the LFS).
However, we received demand from stakeholders to continue to collect variables to meet their needs. This feedback meant that options to reduce the length of the survey were minimal given the limited time remaining before the CDCTP programme and funding ended and the lack of alternative online surveys to house the questions. The fallback position was to transition remaining question blocks on the LFS relatively rapidly onto the TLFS. This became the final design of the first iteration of the TLFS which went live in October 2023.
In 2024, with an unacceptable level of missingness and drop-out on the survey evident in six months of testing, the UK Statistics Authority Board formally endorsed the development and testing of a significantly shorter TLFS labour market “Core” survey. The revised survey design was approved by the Board in March 2025, and the shortened TLFS survey went live on 7 July 2025 (with other design improvements due to be implemented over the next 6 months).
To what extent were stakeholders engaged at the start of the process?
During 2016-18, as the earliest design research was completed, existing LFS stakeholders across government were kept up to date with regular meetings. This included detailed justifications for conducting the research and implications for the overall design of the new survey. This early engagement highlighted that this was a new transformed survey rather than an updated version of the current LFS for online use. Findings from the research were also shared widely via social research industry seminars and conferences and formally published at a later date.
In 2018-19, as part of the initial IPACS design and development, an extensive stakeholder engagement exercise took place with LFS users across central government department, including the Bank of England, HM Treasury and local government interest groups. Requirements for a transformed survey were gathered via a template which was completed by all major LFS stakeholders across government. They were asked which specific LFS variables they currently used, (and hence needed to be included in the transformed survey) what outputs these fed into, and what other survey design requirements they had (e.g. longitudinal data).
This user engagement at the start of the process was maintained at a working level throughout the life of the project. This included continued regular briefings, published guides and information. For example, in December 2023 and January 2024, as the TLFS parallel run was initiated in full, briefings and updates took place with over a hundred users of the LFS across government.
In light of the substantial challenges that labour market statistics faced, and the considerable impact that this was having on key users, the overall approach to stakeholder and user engagement with labour market statistics was significantly improved in 2024. The new approach also looked to address identified limitations with how critical users had been engaged with the TLFS to date.
The key elements of the new approach included:
- A new Labour Market Technical Group chaired at a senior level and meeting monthly, comprising representatives from HMT, the Bank of England, Office for Budget Responsibility (OBR), Department for Work and Pensions (DWP) and Department for Business and Trade (DBT). This provides a space for robust technical dialogue, constructive challenge and meaningful engagement.
- A new independent Stakeholder Advisory Panel chaired by Professor Jonathan Portes with representatives from Government, Academia and Think Tanks. This provides independent advice on the production, publication, uses and applications of labour market statistics and their technical aspects.
- A commitment to regular external updates that combined updates on LFS and TLFS and addressed recommendations from the Office for Statistics Regulation (OSR).
- New opportunities for a wider set of users to engage including public webinars and the UK Statistics Assembly.
- A Household, Socioeconomic and Local Technical Group, established in 2025, that convenes monthly to inform and assure our approach to data collection, requirements, prioritisation and engagement for the wider household and socioeconomic data collected by the TLFS. It has a diverse membership, including users from the devolved governments, central and local government, and think tanks.
This engagement approach was integral to informing, assuring and endorsing the improved TLFS design approved by the Board in March 2025, and will continue to be critical as we move towards transition from the LFS to TLFS.
Who was the Senior Responsible Officer (SRO) for the TLFS?
As detailed in Annex A, the TLFS as a programme is a relatively recent creation. The relevant SROs who have contributed to the history of the TLFS across the accompanying survey development and statistical output development projects are as follows.
| Year | Survey and Processing Development | Statistical Output Development |
|---|---|---|
| 2016 | Data Collection Transformation Programme (DCTP) – Pete Benton, Director of Data Collection | |
| 2017 | ||
| 2018 | ||
| 2019 | Census and Data Collection Transformation Programme (CDCTP) – Iain Bell, Director General and Deputy National Statistician (Population and Public Policy) | |
| 2020 | ||
| 2021 | Donna Leong, Director of Economic Statistics Change and SRO of the Ambitious, Radical, Inclusive Economic Statistics (ARIES) Programme | |
| 2022 | Census and Data Collection Transformation Programme (CDCTP) – After Iain Bell’s departure, Pete Benton briefly stepped in to the SRO role before Ruth Studley was recruited as Director of Population Transformation. | Jason Zawadzki, Director of Economic Statistics Change and SRO of the ARIES Programme |
| 2023 - April 2024 | Ruth delegated to Alex Lambert, Director of Survey Operations. | |
| April 2024 - April 2025 | Survey Enabling Programme – Philippa Bonay, Director of Operations - | |
| April 2025 - onwards | Philippa delegated to Alex Lambert, Director of Social Surveys | |
Programme and Portfolio office
Can you set out in writing what the responsibilities of the programme and portfolio office are (and, if relevant, whether the role of the office has changed over time)?
Can you clarify who has filled this post since 2020?
The roles and responsibilities outlined should be seen in the context of the hierarchy of roles in Annex D. The Portfolio Management team in the ONS supports the portfolio on behalf of the ONS Executive Committee [ExCo]. Prior to a streamlining of governance structures in May 2025, the roles and responsibilities ascribed to ExCo were carried out by a sub-committee named the Portfolio Investment Committee [PIC] which reported to ExCo. These responsibilities have now been absorbed into ExCo.
The ONS portfolio includes significant change programmes required to deliver the UKSA Strategy and ONS Priority Outcomes. The team is part of the Planning and Portfolio Management Division within the Finance, Planning and Performance Directorate. From 2020 – April 2025 Megan Cooper was the Deputy Director for this division; at present James O’Brien occupies the post.
The Portfolio Management team monitors and analyses the portfolio and its constituent programmes on behalf of ExCo, providing regular updates and insights with respect to six key performance indicators: milestones, resource, finance, risks and issues, dependencies, and benefits. As well as regular reporting on the health of the portfolio, the Portfolio Management team offers the following services to Programmes, Projects and Project Management Offices across the ONS:
- Portfolio Assurance – the Portfolio Management team provide second line assurance to SROs on the portfolio via regular monitoring, and an independent perspective on programme performance to ExCo. The team provides expert advice to governance bodies, SROs, and project delivery professionals on governance and assurance, in line with government Project Delivery standards. The portfolio management team also carries out critical friend reviews on behalf of SROs and programme health checks for all programmes on the portfolio (the latter being mandated in 2024 following an internal audit recommendation). Programme Lifecycle Management has been mandated since 2023, and since then three non-Government Major Projects Portfolio (GMPP) programmes under the portfolio have undergone gateway reviews. The Portfolio Assurance Team is actively involved in the planning of National Infrastructure and Service Transformation Authority (NISTA) assurance reviews, as well as coordinating assurance support for wider Government departments.
- Portfolio Governance and Risk – the Portfolio Management team reports portfolio-level risks on behalf of ExCo, identifying and sharing emerging themes across the portfolio for insight and awareness. The team also provides risk support and guidance directly to programmes including facilitating risk identification workshops, reviewing existing risk profiles and governance structures, and provision of risk and issue templates for board packs and highlight reports.
- Business Case Management – the Portfolio Management team coordinates key-holder and wider organisational assurance of business cases on behalf of ExCo to support investment decisions. Through a sub-group of ExCo (Investment and Delivery Assurance Group [IDAG]), the team facilitates awareness and encourages challenge on deliverability and value for money of investments. They also provide advice and guidance to those developing business cases.
- Benefit Management – the Portfolio Management team provides guidance, advice, and leadership on benefits management approach.
- Programme and Project Management lifecycle management – the Portfolio Management team provides advice, guidance and templates to the programmes related to all aspects of the project and programme lifecycle.
The ONS implements a tiered approach to the portfolio based on complexity and strategic value that determines the optimum governance and controls for delivering change successfully, consistent with HM Government’s Teal Book guidance. To be on-boarded onto the ONS portfolio, change activities must have an approved business case that sets out the strategic value and alignment to the ONS Priority Outcomes. Business cases are expected to include a Risk Potential Assessment setting out the level of risk associated with the change which contributes to the decision to on-board the programme to the ONS portfolio. As part of the ONS portfolio, programmes/projects are made more visible to executive governance committees and the Authority Board via monthly reporting of key management information and insights. They are also provided with additional support from the portfolio team when it comes to assurance.
Aside from its primary focus supporting the ONS portfolio, the portfolio management team also provides some support to projects and programmes outside of the portfolio, providing tools and templates to support the complete programme/project lifecycle, signposting directorates to support available from start to finish of their programme journey. The function also undertakes an annual maturity assessment to support continuous improvement in line with the Project Delivery Functional Standard and Teal Book. This assessment is the basis for an action plan that forms the team’s continuous improvement goals for the year.
Integrated Data Service (IDS)
Can you set out costs of the IDS since 2020 in writing?
The Integrated Data Service (IDS) was being delivered by the Integrated Data Programme (IDP), a cross-government programme funded by HM Treasury through a ring-fenced budget. For the financial years 2020/21 to 2024/25 inclusive, the total expenditure recorded against this ring-fenced budget was approximately £223.7million. (This figure was rounded to £224 million and summarised as ‘expenditure to date’ when provided to the Committee as oral evidence on 8 July 2025.) The annual breakdown is as follows
For the financial year 2020/21, the total expenditure of the IDS/IDP ring-fence was approximately £15.5 million.
For the financial year 2021/22, the total expenditure of the IDS/IDP ring-fence was approximately £38.8 million.
For the financial year 2022/23, the total expenditure of the IDS/IDP ring-fence was approximately £56 million.
For the financial year 2023/24, the total expenditure of the IDS/IDP ring-fence was approximately £59.1 million.
For the financial year 2024/25, the total expenditure of the IDS/IDP ring-fence was approximately £54.3 million.
Pete Benton was appointed SRO of this programme by ONS and the (then) IPA in 2024, but a separate DG for the IDS was also appointed. Can you please clarify who is responsible for what?
The decisions for these appointments were made by the former National Statistician, Sir Ian Diamond.
Nigel Green was appointed to undertake a specific IDP role on a two day a week contract, adding programme delivery leadership. Nigel has a track record of successful major programme delivery from elsewhere in government with a digital background. Pete Benton’s background as a statistician allowed him to lead on developing the service in a way that could meet the analytical demands of users.
Pete and Nigel’s collaboration is best described as a shared leadership model, with Pete focusing on strategy and stakeholder engagement, and Nigel driving delivery and programme execution.
Nigel Green could not be SRO for the programme as Infrastructure and Projects Authority (IPA)/National Infrastructure and Service Transformation Authority (NISTA) require that this role only be undertaken by an appointed Civil Servant, and not by those in a contractor role. In addition, the role that Pete Benton was asked to undertake (to cover for a Director General’s sick leave) was broader than just the IDP, with leadership responsibilities across the Data Capability Group. The Data Capability Group includes the IDP and IDS as well as the Digital Services, and the Data Growth & Operations directorates.
Can you confirm if there were any major data programs which now will not be able to proceed because of the closure of the IDS program?
While the ONS does not use the term ‘major data programme’ in relation to the IDS analytical project pipeline, the pipeline of projects has been assessed on their alignment to ONS priorities and their potential impact to Government Missions. This has resulted in a number of IDS project applications that do not align to ONS priorities being declined, including seven project applications that could support government missions on employment, health, crime and economy. Where we deem a project to have high potential impact to Government Missions these will be considered on a case-by-case basis by ExCo as part of our exemption process, which is currently in development.
Linked data represents a highly valuable and underutilised asset to government. Secure access to safely linked data, as enabled by the IDS, creates new opportunities for highly impactful analysis, unlocking the insights needed to shape policy and lead to better outcomes for citizens.
The IDS has responded to government demand and focussed on linkages that unlock important policy questions, such as the linkages between health and labour market data; the linkage between properties (including energy usage) and individuals; and the linkage of business data, including links to employees. The IDS was previously focused on supporting analytic projects that align to the government’s central missions, such as analysis of healthcare interventions and labour market activity, and research into attributes and geographic distributions of people and households living in poverty.
Restricting access to the linked data / data linkage assets will limit the potential of this valuable government resource. The decision to focus the assets on ONS priorities, where they can contribute to our core statistics, means that the IDS is only accepting new applications for external use by strict exception.
The ONS is working closely, and on a case-by-case basis with its partners in academia and across government to fully understand and mitigate the impact of restricted access. In particular, the ONS is:
- Working with Administrative Data Research UK (ADR UK) to support the continued preparation of research-ready data, access to which could be provisioned to academic researchers via the Secure Research Service;
- Working with DSIT to explore options for the work of the IDS to be integrated into the National Data Library’s Kickstarter Programme; and
- Developing an exemption process, whereby access to linked data already held by ONS could be provided to external analysts in limited instances – where analysis is assessed to be of high value to government; with an underpinning cost-recovery model; and where this activity does not distract from the ONS’ core organisational priorities.
Office for National Statistics
August 2025
Annex A: TLFS timeline
| Year | Survey and Processing Development | Statistical Output Development |
|---|---|---|
| 2016 | Original design, stakeholder engagement and research work under the Data Collection Transformation Programme (DCTP) | |
| 2017 | ||
| 2018 | ||
| 2019 | Initial Integrated Population and Characteristics Survey (IPACS) design and development within the Census and Data Collection Transformation Programme (CDCTP). Renaming of IPACS into the Labour Market Survey (LMS). | |
| 2020 | Launch of the partially completed LMS survey during the pandemic as the Labour Force Survey (LFS) response rates dropped significantly due to lockdown. While the LMS was launched in its half-finished state, further development was deprioritised due to Census 2021 and the Covid Infection Survey. | |
| 2021 | Re-start and development of the LMS post Census 2021. Remaining survey questions were added to the online version and telephone and ‘knock-to-nudge’ modes added. Parallel run against LFS starts October 2023. | Creation of the accompanying Labour Market Transformation project within the Ambitious, Radical, Inclusive Economic Statistics (ARIES) Programme within ESEG to manage the transition of labour market outputs from LFS to LMS. |
| 2022 | LFS and TLFS quality compared in April 2024 – further work required due to quality issues. | |
| 2023 | ||
| 2024 | End of CDCTP programme, TLFS project moved to a new Survey Enabling Programme. | |
| 2025 | Development of shorter TLFS completed and formal decision in March 2025 to create a single TLFS programme to implement the improved TLFS design. The new programme encompasses previous work on TLFS survey and processing design and Labour Market Transformation. | |
Annex D: Roles and Responsibilities for Project Delivery
- Accounting Officer – accountable to Parliament for how public money is spent by ONS. Chairs the Executive Committee.
- Senior Responsible Owner (SRO) – has delegated authority (via an SRO appointment letter) from the Accounting Officer, and in the case of the Government Major Projects Portfolio (GMPP), from HMT to run projects/programmes according to best practice and in line with HMT guidance on the use of public money. The SRO of a GMPP programme is accountable to Parliament for the performance of the programme and ensuring it meets the needs of the business as set out in the business case.
- Director of Finance, Planning and Performance is responsible for ensuring that public money is safeguarded and used appropriately and efficiently, advising the Accounting Officer in this regard.
- The Planning & Portfolio Management Division supports executive governance by advising on the processes and structures to enable successful portfolio management and associated decision making. They present the monthly portfolio scorecard along with insights drawn from trend analysis and lessons learned. They provide project delivery support to the organisation aligned to conventional project delivery standards. Regular engagement with project delivery staff takes place to ensure these processes and practices are understood.
- Key Holders support decision making by providing assurance on business cases at appropriate staged gates.
- Programme Management Offices are set up to support the SROs of significant programmes. These are situated within the directorates from which the programmes are being delivered.