The purpose of this note is to provide the Methodological Assurance Review Panel (MARP) and the Office for National Statistics (ONS) with a stocktake of emerging issues in our review of the methodology underpinning the Transformed Labour Force Survey (TLFS) as well as the current ability of the TLFS to be a viable replacement for the Labour Force Survey (LFS) as the main ONS source of UK labour market data.

We started this review following a meeting with ONS representatives on 22 April. As of 14 May we have had 7 intensive video consultations with relevant members of the ONS Methodology Group responsible for TLFS design and evaluation as well as with key members of ONS Labour Market Branch responsible for TLFS implementation. We are extremely grateful to everybody that we have interacted with in this process for their willingness to share their knowledge of TLFS and LFS methodology and the issues in transferring from one data collection platform to the other.

The LFS is the most important source for labour market data collected by the ONS, and its design and implementation has remained consistent since the late 90s. However, as with household surveys all over the world, LFS sample non-response and associated longitudinal attrition is steadily increasing, to the extent that there is now doubt about its medium to long term viability. This issue was recognised some years ago by the ONS and a program was initiated to introduce a new survey, the TLFS.

The design of the TLFS represents a significant departure from the LFS:

• The LFS is an interviewer-based household data collection with a longitudinal 5 wave sample structure corresponding to households being interviewed every quarter for 5 consecutive quarters.
• The TLFS has the same 5 wave longitudinal structure but collects most of its labour force data via a self-completed online questionnaire, and its sample design involves a much larger wave 1 sample than the LFS, with a smaller pre-selected subsample of wave 1 households that are followed up to provide longitudinal information in waves 2 – 5.

In its most recent form, the TLFS has now been used to provide headline person-level labour force estimates covering two quarters (OD23 and JM24), comprising 6 sets of rolling monthly estimates. It is expected that the first TLFS flows data set across those two quarters will be available in early July 2024, with estimates for the AJ24 quarter becoming available afterwards. A key issue that we were asked to comment on was whether it would be reasonable for the ONS to aim to replace LFS estimates for this quarter by TLFS estimates and to effectively “switch off” the LFS from July 2024. We do not believe this transition strategy to be advisable.

There are four major issues that influence this opinion, and they relate to the four main data sets created from LFS:

• By July 2024 there will be at most three quarters of TLFS data available. In what we have seen so far (OD23 and JM24 quarters parallel running) the TLFS estimates and the LFS estimates exhibit different levels and dynamics. Adding the AJ24 quarter is unlikely to stabilise this comparison and will not be sufficient to generate a robust set of back-casted labour force headline estimates comparable with corresponding LFS estimates. To properly model any differences in seasonal behaviour a minimum of 4 quarters parallel running is needed. Parallel running of 5 quarters would allow full comparison of the OD23 wave 1 cohort completing a full data collection cycle.
• No quarterly flows estimates have yet been produced from TLFS, though the OD23 – JM24 flows estimates are due in July following the release of LFS flows estimates in May. No QA on the comparison of these flows estimates will be possible before July. It is unlikely that even if TLFS flows appear reasonable there will be sufficient confidence in their robustness to switch from the LFS to the TLFS for subsequent flows.
• The TLFS methodology for producing an annual labour force data set has yet to be finalised, so we do not know whether it will lead to an acceptable data set for the fine level spatial analyses that will be carried out using it.
• The TLFS methodology for producing a household-based dataset has yet to be finalised, and part of the uncertainty with respect to household-based weighting relates to accounting for within household drop-out in the TLFS online data collection.

These uncertainties imply further parallel running the TLFS and the LFS may be necessary if the ONS wishes to provide its users with an acceptable replacement for the LFS going forward, and a solid assessment of the differences in the results produced by the two surveys going backward that will allow users to transition their own LFS-based data sets to comparable TLFS-based data sets with a minimum of pain.

Some more detailed observations are:

• There remain issues for the person-level weighting and non-response attrition adjustments in the TLFS. The weighting methodology used by the TLFS is more complex than that used by the LFS since
  • wave 1 in the TLFS is very large, with subsampling for subsequent waves;
  • respondents are allowed to provide data for self-identified reference weeks within a 4 week timeframe following their selection;
  • non-respondents to wave 1 are allowed to respond to subsequent waves.
• At present it is unclear whether it can adequately compensate for some of the observed labour force status and health status differences in TLFS response patterns.
• The interaction between within record missing data imputation and weighting to account for person-level non-response / attrition in the TLFS has not been investigated.
• Household level weighting for the TLFS remains unimplemented and untested.
• There remain questions about the design of the online TLFS questionnaire and whether this is a contributing factor to the observed interaction between labour force status, choice of reference week and sample attrition.

Some concluding remarks looking forward:

• The original concept of the TLFS remains very sensible – i.e., a short online survey collecting data on core LFS variables with the capacity for designed modular additions to capture additional labour force information.
  • We strongly recommend that it remains the long-term design objective of the TLFS.
• The reality of the current TLFS implementation is somewhat different:
  • It seems to be an attempt to capture online all the information that is currently captured via a person-level interview by the LFS in an attempt to meet a range of complex user needs.
  • This has resulted in downstream attrition issues plus the need for additional data capture strategies (telephone, Knock to Nudge) which make processing the TLFS rather complicated, and has effectively negated the flexibility that was a crucial part of how the TLFS was originally conceived.
• At this stage it is unclear whether the complexity of the current TLFS is only necessary for the parallel running period or whether users will require it going forward.
  • What is clear is that (most of) the projected TLFS gains over the LFS (simplicity, flexibility, improved response, less measurement error, larger sample, more stable person-level labour force dynamics) are harder to achieve and maintain with the current length of the online questionnaire.

Ray Chambers, FASSA
Honorary Professor of Statistical Methodology
University of Wollongong and Australian National University

James Brown
Professor of Official Statistics
University of Technology Sydney

16 May 2024