The use of agreement rates between individual data on the Census and an alternative source as a reliable metric for respondent error relies on the alternative source:

• providing data on the same concept as the census;
• being sufficiently accurate in collecting individual data relating to Census day;
• avoiding the tendency to record the same errors that appear on the census (for example, by using a different mode of collection);
• providing data on a representative sample of the population.

Option 1: Face-to-face Interview CQS

This was the approach adopted in 2011. The methodology for that survey is described in the 2011 CQS Report. The approach was tested in the CQS Pilot in January/February 2020 and no major concerns were identified.

The CQS is designed to collect information relating to the same concepts as the census (in effect, asking the same questions) and the use of trained interviewers with supporting information to help respondents should ensure sufficient accuracy of response. The mode of collection is independent of the census.

Whilst the initial sample can be designed to be representative of the population of interest (that is, individuals on census returns) and post-stratification or calibration can be used to adjust for differential response between demographic groups (defined with reference to characteristics collected on the census), there remains the possibility that differential response within those groups may affect the accuracy of the estimated agreement rates as a measure of respondent error.

Option 2: Telephone Interview CQS

This approach is similar to Option 1 except that the interviews would be conducted via telephone rather than face to face. This approach was also tested in the CQS pilot and, again, no major concerns were identified.

We have no evidence on the quality of individual data collected via the telephone approach compared with face-to-face interviews but note that interviews via each method are conducted by trained interviewers with specific briefing on the survey and with guidance on helping respondents answer the survey accurately. Telephone surveys are used as the data collection method for various National Statistics. The Labour Force Survey QMI, for example notes the possibility of measurement error but does not seek to quantify it nor distinguish between modes of collection. It would seem reasonable to assume that the difference in accuracy between telephone and face-to-face interview with an individual will be small compared to the inaccuracy present on census returns.

Whilst a very small amount of census data may be collected by telephone, this approach would be, for practical purposes, an independent mode of collection to the main census.

We have identified a potential issue with the representivity of the CQS sample due to the use of a telematching service to obtain a telephone number relating to the address appearing on the census return. At present, these numbers are restricted to landline numbers. It is reasonable to assume that the use of landline numbers is not constant across demographic groups and that a simple sample selection would result in overcoverage of some groups. Further, the pilot survey indicated that retired individuals were disproportionately likely to respond to the survey. Both these issues can be mitigated by oversampling of some groups to ensure that the final achieved sample is adequately representative and by post-stratification or calibration of the estimates.

Option 3: Other Surveys

A research project was started in October 2019 to investigate whether an alternative to running a separate CQS was possible. It succeeded in identifying an approach using existing ONS surveys (Census Coverage Survey (CCS), surveys linked through the Census Non-Response Link Study (CNRLS), and an additional module on the Opinions and Lifestyle Survey (OPN)) which would allow the linked-individual comparisons required for the estimation of respondent error. Annex B summarises this approach.

By primarily using data that will already be linked to the Census for other purposes, we derive additional value without incurring additional cost. Making use of existing data sources wherever possible supports the goals of the current transformation of ONS statistics.

Furthermore, the CCS and CNRLS surveys offer a far larger sample size than could be gathered – or successfully linked – as part of a wholly separate CQS project.

The information to be linked to the census under this approach can generally be expected to be of good quality as it is collected by trained interviewers with appropriate supporting information as with the previous approaches. There are some instances where the data collected on other surveys relates to a slightly different concept or is collected in a different way which would complicate the calculation or interpretation of some agreement rates. Data collected via an OPN module would be collected through a web self-completion which would not be independent of the mode of collection of the main census (but, as with other approaches, would identify errors caused through proxy responses).

More information on this approach is provided in Annex B. It was not possible, however, within the scope of the project and available resources to test this approach on existing sources and demonstrate that it would provide the same quality or better as a dedicated CQS.

The strengths and weaknesses of the three approaches are summarised below.

Table 1: Approaches to Measuring Respondent Error

Quality Assurance

As the results of the CQS would be likely to be available only at a late stage in the QA process, it is difficult to envisage any adjustment to the Census data as a direct result of the CQS. This is seen as information to be used to help users understand uncertainty in the Census results, rather than being a tool in the QA itself.

Public Commitments

Paragraphs 4.97-4.99 of the Census White Paper state that a CQS will be carried out after the Census. There are no other commitments recorded on the Census Commitments database.

Demand from Data Users

The Census Historical Data team have no record of any interest or queries relating to the 2011 CQS. The 2021 Census Outputs and Dissemination team have confirmed that no reference to the CQS has been made in the consultation and engagement on the 2021 outputs. The web analytics team are unable to provide data on hits/downloads of the CQS page and report, due to the archiving of content and re-implementation of the website in 2016. A request for CQS data was received from the Welsh Government and we are checking whether a similar request is likely for 2021 data.

NS Accreditation

The three Census UKSA Assessment reports from 2011 provide no indication that the CQS was a factor in the decision to award National Statistics status. The only reference found is in Paragraph 3.27 of Report 1 in the context of ONS looking to harmonise its approach to the CQS with NRS and NISRA.

The first report provided to UKSA for the 2021 Assessment repeats the White Paper’s comments on the plan to conduct a CQS in 2021.

International Good Practice

Both NRS and NISRA conducted a CQS following the 2011 Censuses. The UNECE report Recommendations for the 2010 Census of Population and Housing refers (p193) to the possibility of running a post-census survey to measure content error but does not identify any countries carrying out such a survey.

Other Uses

The Census Question and Questionnaire Design team have indicated that the wider Social Survey Transformation team will be likely to have an interest in respondent error rates. This information can be used to help iterate and improve upon the design of corresponding questions in other surveys.

It is noted that analysis of the accuracy of responses to a survey-based census may be especially useful in the context of the 2021 Census, as this may help inform the discussion around potential future admin-first models.

This Annex describes the results of a research project which developed a proposal for measuring respondent error in the 2021 Census with running a separate CQS. It does not describe the detail of how this proposal was derived: in summary, we identified alternative sources for data for each census question and evaluated these with regard to how closely the response for an individual should correspond to the true value at census day (for example, taking account of any difference in question design or time differences in when the question was asked) and whether it was practicable to link the alternative source to the census responses in order to calculate agreement rates.

The proposed design measures around 25% of census questions using Census Coverage Survey (CCS) data; around 50% using Labour Force Survey (LFS) data (via the Census Non-Response Link Study (CNRLS)); and the remaining 25% via a custom Opinions and Lifestyle (OPN) survey module, collected over the four months spanning Census day. The preferred source for each question is shown in the tables at the end of this Annex.

Broadly speaking, the proposed design sources questions from the CCS as a priority. By its nature, questions shared with the CCS are generally exact matches in terms of design. In addition to its large sample size, the CCS boasts high quality linkage and a shared reference date, which minimizes time sensitivity concerns. Methods of linkage would be taken into account to ensure that agreement rates were not incorrectly inflated through links only being made to records where responses to the census and CCS agreed. CCS data is expected to be of high quality (being collected through face-to-face interviews though proxy data for other members of the household will be collected)

LFS data – via the CNRLS – is the second resort, due to its broad question coverage and similarly vast sample size. Corresponding questions are not always identical between the two sources and there will be some mismatch in timing between the two sources (LFS responses would be taken from the two months before and after Census day). Where timing differences are seen to have a material effect on agreement rates (for example, with the activity last week question) we will investigate modelling the agreement rate as a linear regression model with time from Census day as the explanatory variable (allowing the estimation of the agreement rate that would apply at Census day). LFS data is collected both through face-to-face interviews and telephone interviews and proxy data is collected.

Only questions unavailable via any existing source were slated for the OPN module. These include questions on sexual orientation and gender identity, where there will be particular interest in the accuracy of the census responses

The OPN will have a relatively small sample size of around 4,000 respondents but this should be sufficient to achieve the target accuracy levels in 2011 (agreement rates with a maximum margin of error of +/- 2 percentage points) for all questions other than  second address type and English ability. The latter questions sit on very uncommon routing branches which means that the issue cannot easily be resolved by increasing the overall sample size. No alternative existing sources are available for these questions; a lower grade analysis using the responses we can muster is still the best option. This shortage of frequency was also the case in 2011[1], and would also be an issue for any dedicated CQS design in 2021. Furthermore, there are several other questions that did not achieve sufficient frequency for analysis in 2011, but which will be analysable under this design due to being sourced via CCS and LFS with their exceptionally large samples.

The OPN is now is online-first with telephone follow-ups for non-responders so will not primarily be a different mode of collection to the main census. It does not collect proxy data.

Building our design with dependencies on the CCS, CNRLS, and OPN necessarily incurs risks and uncertainty. Notably, because the CNRLS has itself been developing in parallel with this project, some details – such as the format and extent of their matching resource – will still to be determined as of the time of the recommendation report.

Table A1: Household Questions

Table A2: Household Questions