This paper describes three approaches to measuring respondent error in the 2021 Census and invites comments from Panel members to guide the final recommendation on the preferred approach.
A Census Quality Survey (CQS) was used in the 2011 Census to estimate respondent error for each individual census question. Following the Census itself, a sample of about 12 thousand household residents was asked the same questions a second time, in a separate survey. This allowed agreement rates to be calculated by identifying the fraction of respondents whose responses differ between census and survey. Respondent error is likely to be more of an issue for the census than for many ONS survey sources as the census is not interviewer administered and as there are expected to be a large proportion of proxy responses (information provided on behalf of someone else in the household). The results of the 2011 CQS are published in the 2011 CQS report and identified uses of the CQS results are described in Annex A.
The 2011 survey (and a similar survey ahead of the 2001 census) was conducted as face-to-face guided interviews. This format is considered the “gold standard” for data quality. Disagreements are therefore interpreted as respondent error on the census form.
The default assumption during preparations for the 2021 Census has been that we will once again run a CQS. The Census White Paper reflects this intention and a pilot survey was run in early 2020.
We have also conducted research into whether an alternative approach using existing administrative or survey sources could provide the same or better results with advantages of lower cost and respondent burden.
In September 2020 we will seek a decision from the PPP Transformation Board on the preferred approach for measuring respondent error. This note outlines the three broad approaches that will be presented. We invite comments from Panel members on those approaches. This paper does not present detailed information on how each approach would be implemented.
The use of agreement rates between individual data on the Census and an alternative source as a reliable metric for respondent error relies on the alternative source:
- providing data on the same concept as the census;
- being sufficiently accurate in collecting individual data relating to Census day;
- avoiding the tendency to record the same errors that appear on the census (for example, by using a different mode of collection);
- providing data on a representative sample of the population.
Option 1: Face-to-face Interview CQS
This was the approach adopted in 2011. The methodology for that survey is described in the 2011 CQS Report. The approach was tested in the CQS Pilot in January/February 2020 and no major concerns were identified.
The CQS is designed to collect information relating to the same concepts as the census (in effect, asking the same questions) and the use of trained interviewers with supporting information to help respondents should ensure sufficient accuracy of response. The mode of collection is independent of the census.
Whilst the initial sample can be designed to be representative of the population of interest (that is, individuals on census returns) and post-stratification or calibration can be used to adjust for differential response between demographic groups (defined with reference to characteristics collected on the census), there remains the possibility that differential response within those groups may affect the accuracy of the estimated agreement rates as a measure of respondent error.
Option 2: Telephone Interview CQS
This approach is similar to Option 1 except that the interviews would be conducted via telephone rather than face to face. This approach was also tested in the CQS pilot and, again, no major concerns were identified.
We have no evidence on the quality of individual data collected via the telephone approach compared with face-to-face interviews but note that interviews via each method are conducted by trained interviewers with specific briefing on the survey and with guidance on helping respondents answer the survey accurately. Telephone surveys are used as the data collection method for various National Statistics. The Labour Force Survey QMI, for example notes the possibility of measurement error but does not seek to quantify it nor distinguish between modes of collection. It would seem reasonable to assume that the difference in accuracy between telephone and face-to-face interview with an individual will be small compared to the inaccuracy present on census returns.
Whilst a very small amount of census data may be collected by telephone, this approach would be, for practical purposes, an independent mode of collection to the main census.
We have identified a potential issue with the representivity of the CQS sample due to the use of a telematching service to obtain a telephone number relating to the address appearing on the census return. At present, these numbers are restricted to landline numbers. It is reasonable to assume that the use of landline numbers is not constant across demographic groups and that a simple sample selection would result in overcoverage of some groups. Further, the pilot survey indicated that retired individuals were disproportionately likely to respond to the survey. Both these issues can be mitigated by oversampling of some groups to ensure that the final achieved sample is adequately representative and by post-stratification or calibration of the estimates.
Option 3: Other Surveys
A research project was started in October 2019 to investigate whether an alternative to running a separate CQS was possible. It succeeded in identifying an approach using existing ONS surveys (Census Coverage Survey (CCS), surveys linked through the Census Non-Response Link Study (CNRLS), and an additional module on the Opinions and Lifestyle Survey (OPN)) which would allow the linked-individual comparisons required for the estimation of respondent error. Annex B summarises this approach.
By primarily using data that will already be linked to the Census for other purposes, we derive additional value without incurring additional cost. Making use of existing data sources wherever possible supports the goals of the current transformation of ONS statistics.
Furthermore, the CCS and CNRLS surveys offer a far larger sample size than could be gathered – or successfully linked – as part of a wholly separate CQS project.
The information to be linked to the census under this approach can generally be expected to be of good quality as it is collected by trained interviewers with appropriate supporting information as with the previous approaches. There are some instances where the data collected on other surveys relates to a slightly different concept or is collected in a different way which would complicate the calculation or interpretation of some agreement rates. Data collected via an OPN module would be collected through a web self-completion which would not be independent of the mode of collection of the main census (but, as with other approaches, would identify errors caused through proxy responses).
More information on this approach is provided in Annex B. It was not possible, however, within the scope of the project and available resources to test this approach on existing sources and demonstrate that it would provide the same quality or better as a dedicated CQS.
The strengths and weaknesses of the three approaches are summarised below.
Table 1: Approaches to Measuring Respondent Error
|CQS conducted through field interviews||· Consistent with previous approach|
· Tested in pilot.
· Highest quality data
· Simple to analyse and report.
|· Expensive (data collection costs estimated at £1.1m).|
· Respondent burden
· Requires face-to-face interviews with higher risk of operational problems or public unacceptability due to COVID19
CQS conducted through field interviews
· Expensive (data collection costs estimated at £1.1m).
· Respondent burden
· Requires face-to-face interviews with higher risk of operational problems or public unacceptability due to COVID19
CQS conducted through telephone interviews
|· Tested in pilot|
· Good quality data
· Simple to analyse and report
· Low risk from COVID19.
· Reduced data collection costs (£0.2m)
|· Respondent burden|
· Greater risk of unrepresentative sample
|Alternative approach: existing surveys supplemented by OPN module||· Low data collection costs (£40K)|
· Consistent with ONS strategy
· Larger sample sizes for many questions allowing more detailed analyses
· Low risk from COVID19
|· Complex to analyse and report|
· Less reliable data for sexual orientation and gender identity
· Some increase in requirements on other teams.
· Increased development costs
· Not tested with increased risk to quality of results
· Some risk of criticism for not meeting White Paper commitment or during NS Accreditation process
ANNEX A: Uses Made of the 2011 CQS Results
As the results of the CQS would be likely to be available only at a late stage in the QA process, it is difficult to envisage any adjustment to the Census data as a direct result of the CQS. This is seen as information to be used to help users understand uncertainty in the Census results, rather than being a tool in the QA itself.
Paragraphs 4.97-4.99 of the Census White Paper state that a CQS will be carried out after the Census. There are no other commitments recorded on the Census Commitments database.
Demand from Data Users
The Census Historical Data team have no record of any interest or queries relating to the 2011 CQS. The 2021 Census Outputs and Dissemination team have confirmed that no reference to the CQS has been made in the consultation and engagement on the 2021 outputs. The web analytics team are unable to provide data on hits/downloads of the CQS page and report, due to the archiving of content and re-implementation of the website in 2016. A request for CQS data was received from the Welsh Government and we are checking whether a similar request is likely for 2021 data.
The three Census UKSA Assessment reports from 2011 provide no indication that the CQS was a factor in the decision to award National Statistics status. The only reference found is in Paragraph 3.27 of Report 1 in the context of ONS looking to harmonise its approach to the CQS with NRS and NISRA.
The first report provided to UKSA for the 2021 Assessment repeats the White Paper’s comments on the plan to conduct a CQS in 2021.
International Good Practice
Both NRS and NISRA conducted a CQS following the 2011 Censuses. The UNECE report Recommendations for the 2010 Census of Population and Housing refers (p193) to the possibility of running a post-census survey to measure content error but does not identify any countries carrying out such a survey.
The Census Question and Questionnaire Design team have indicated that the wider Social Survey Transformation team will be likely to have an interest in respondent error rates. This information can be used to help iterate and improve upon the design of corresponding questions in other surveys.
It is noted that analysis of the accuracy of responses to a survey-based census may be especially useful in the context of the 2021 Census, as this may help inform the discussion around potential future admin-first models.
ANNEX B: Measuring Respondent Error Without a Census Quality Survey
This Annex describes the results of a research project which developed a proposal for measuring respondent error in the 2021 Census with running a separate CQS. It does not describe the detail of how this proposal was derived: in summary, we identified alternative sources for data for each census question and evaluated these with regard to how closely the response for an individual should correspond to the true value at census day (for example, taking account of any difference in question design or time differences in when the question was asked) and whether it was practicable to link the alternative source to the census responses in order to calculate agreement rates.
The proposed design measures around 25% of census questions using Census Coverage Survey (CCS) data; around 50% using Labour Force Survey (LFS) data (via the Census Non-Response Link Study (CNRLS)); and the remaining 25% via a custom Opinions and Lifestyle (OPN) survey module, collected over the four months spanning Census day. The preferred source for each question is shown in the tables at the end of this Annex.
Broadly speaking, the proposed design sources questions from the CCS as a priority. By its nature, questions shared with the CCS are generally exact matches in terms of design. In addition to its large sample size, the CCS boasts high quality linkage and a shared reference date, which minimizes time sensitivity concerns. Methods of linkage would be taken into account to ensure that agreement rates were not incorrectly inflated through links only being made to records where responses to the census and CCS agreed. CCS data is expected to be of high quality (being collected through face-to-face interviews though proxy data for other members of the household will be collected)
LFS data – via the CNRLS – is the second resort, due to its broad question coverage and similarly vast sample size. Corresponding questions are not always identical between the two sources and there will be some mismatch in timing between the two sources (LFS responses would be taken from the two months before and after Census day). Where timing differences are seen to have a material effect on agreement rates (for example, with the activity last week question) we will investigate modelling the agreement rate as a linear regression model with time from Census day as the explanatory variable (allowing the estimation of the agreement rate that would apply at Census day). LFS data is collected both through face-to-face interviews and telephone interviews and proxy data is collected.
Only questions unavailable via any existing source were slated for the OPN module. These include questions on sexual orientation and gender identity, where there will be particular interest in the accuracy of the census responses
The OPN will have a relatively small sample size of around 4,000 respondents but this should be sufficient to achieve the target accuracy levels in 2011 (agreement rates with a maximum margin of error of +/- 2 percentage points) for all questions other than second address type and English ability. The latter questions sit on very uncommon routing branches which means that the issue cannot easily be resolved by increasing the overall sample size. No alternative existing sources are available for these questions; a lower grade analysis using the responses we can muster is still the best option. This shortage of frequency was also the case in 2011, and would also be an issue for any dedicated CQS design in 2021. Furthermore, there are several other questions that did not achieve sufficient frequency for analysis in 2011, but which will be analysable under this design due to being sourced via CCS and LFS with their exceptionally large samples.
The OPN is now is online-first with telephone follow-ups for non-responders so will not primarily be a different mode of collection to the main census. It does not collect proxy data.
Building our design with dependencies on the CCS, CNRLS, and OPN necessarily incurs risks and uncertainty. Notably, because the CNRLS has itself been developing in parallel with this project, some details – such as the format and extent of their matching resource – will still to be determined as of the time of the recommendation report.
Table A1: Household Questions
|H||1||(who lives here - not asked on CQS)||N/A|
|H||2||(count of people - not asked on CQS)||N/A|
|H||3||(names of people - used for linkage)||N/A|
|H||4||(visitors - not asked on CQS)||N/A|
|H||5||(count of visitors - not asked on CQS)||N/A|
|H||14||Cars & vans||OPN|
Table A2: Household Questions
|P||(proxy indicator - not asked on CQS)||N/A|
|1||(individual name - used for linkage)||N/A|
|2||Date of birth||CCS|
|5||Sex of partner||OPN|
|7||Second address type||OPN|
|10||Country of birth||LFS|
|11||Date of arrival||LFS|
|12||Intention to stay||CCS|
|13||Address one year ago||OPN|
|23||Impact of illness||LFS|
|33||Activity in last 7 days||CCS|
|35||Looking for work||LFS|
|36||Availability in 2 weeks||LFS|
|37||Waiting to start job||LFS|
|41||(business name - not asked on CQS)||N/A|
|47||Hours per week||LFS|
|48||Method of travel||LFS|
|49,50||Location of work||OPN|