Post MARP note: This paper represents initial thinking on COVID-19 impact when presented to the panel in May 2020. Further information on impact and the statistical design has subsequently been published on the ONS website.

Overview

The impact of Covid19 is being considered within ONS with reference to three scenarios:

• Scenario 1 – Disruption for 3 months but transition back to normal operations by July 2020
• Scenario 2 – Same as scenario 1 with additional unplanned disruption in Autumn/Winter 2020
• Scenario 3 – Same as scenario 2 with additional planned and ongoing disruption

Impact on statistical quality is set out in the annex to this note. With a summary on statistical quality set summarised below.

Overview of Scenarios on Statistical Quality

Impact Scale = 1 (Minor) to 5 (Major)

Impact on Statistical Design Phases

Scenario	Design/Build	Monitor/Counter	Process/Estimate
1 (Total Impact Score = 7)	• Address Frame Quality without a field address check (1) • Economic/Social change impacts Hard to Count (HtC) index (1) • Modelling assumptions changed (1) • Delay in recruiting community engagement officers impacting Target Action Groups (1)	• Reduced ability of Response Chasing Algorithm to effectively identify interventions due to changes in underlying modelling assumptions (1)	• Uncertainty with availability of administrative data so potential impact on ability to identify and adjust (1) • Increased variability in other statistical design elements result in increased variability In resulting estimates (1)
2 (Total Impact Score = 14)	• Address Frame Quality without a field address check (1) • Economic/Social change impacts HtC (1) • Modelling assumptions changed (1) • Delay in recruiting community engagement officers impacting Target Action Groups (3)	• Reduced ability of Response Chasing Algorithm to effectively identify interventions due to changes in underlying modelling assumptions (1) • Field operation capacity and capability is impacted (2)	• Uncertainty with availability of administrative data so potential impact on ability to identify and adjust (2) • Increased variability in other statistical design elements result in increased variability In resulting estimates (2) • Census Coverage Survey (CCS) planning begins to be impacted (1)
3 (Total Impact Score = 29)	• Address Frame Quality without a field address check (1) • Economic/Social change impacts HtC (3) • Modelling assumptions changed (3) • Delay in recruiting community engagement officers impacting Target Action Groups (5)	• Reduced ability of Response Chasing Algorithm to effectively identify interventions due to changes in underlying modelling assumptions (2) • Field operation capacity and capability becomes minimised resulting in major variability and bias in estimates (5)	• Uncertainty with availability of administrative data so potential impact on ability to identify and adjust when our need to adjust is inevitable (4)* • CCS is incapable of delivering a meaning basis for coverage estimation. Without ability to undertake an address listing exercise no missed addresses would be identified (5)*

*Alternative options are available to standard coverage estimation approach (weighted class) but are dependent on alternative data

**Options are available for CCS (delay, running in certain areas and borrowing strength

Annex – Covid19 Impact by Statistical Design Phases

Statistical Design Phase 1 – Design/Build

Area of Statistical Design	Potential Covid Implication	Potential Statistical Quality Impact
Address	• Unable to run a field address check – so increase scale and scope of clerical address check • SSD resource available to increase scale of DART team and Communal Establishment (CE) clerical review • Availability of managers in CEs which may need to respond to requests from the clerical review process to complete the CE address frame. • Availability of administrative data for remaining areas of construction of the CE address frame (Ministry of Defence, Ministry of Justice etc).	• Field address check would have resolved some addresses which DART could not meaning resolution in field (so reducing overall efficiency) • Users maybe concerned about quality without a field address check • Quality of CE address frame is particularly likely to be impacted and is likely to be affected for some types of CE i.e. prisons and armed forces bases • May also impact on ability of field operation to accurately follow up response for student halls resulting in a lower response rate. • Significant reduction in building reduces underlying level of change addresses
Questionnaire	• More challenging to make any late changes (though changes unlikely)	• Questionnaire development complete
Wave of Contact	• Covid scenarios likely to change assumptions used in modelling and extremely difficult to make assumptions about 2021 e.g. field effectiveness maybe impacted by challenges with training staff of making face-to-face contact. • Additional modelling maybe required and live updates to modelling required during operation. • Administrative data availability likely to be impacted (Council Tax) used in prioritising field follow-up	• Potential that assumptions are incorrect in the Field Operations Simulation (FOS), meaning field staff volumes or distribution may be insufficient to reach response rate and variability targets and effectiveness of the Response Chasing Algorithm in optimising response may be reduced. • Impact on field resource availability would disrupt wave of contact model • Reduced availability of administrative data would reduce field efficiency impacting variability across Local Authorities.
Hard to Count (Digital)	• Likely to be a positive impact with increased confidence in online access.	• Increase online response
Hard to Count (Willingness)	• Impact of economic recession will mean change patterns of willingness. • Some areas will be harder to count than in 2011 and changes unlikely to be picked up in time with existing admin data at small area level	• Response chasing will be less accurate with out of date HTC and will result in an increase in variability at small area level (though likely to be minor)
Target Action Groups	• There may be new Target Action Groups that have not yet been identified (i.e. lack of confidence in government, newly unemployed) • Existing Target Action Groups maybe more or less difficult to engage • Closely associated with community engagement so any reduction in their effectiveness will impact directly on TAG strategies.	• Identification of and planning for new groups required and risks lower response in some groups (increasing variability) • Dependency on community engagement officers will directly impact response in community groups (increasing variability)
Market Segmentation	• Potential to change existing segmentation (unlikely to be major)	• Potential inefficiency or reduced effectiveness of comms messaging impacting overall response.
Community Engagement	• Recruitment and training of community engagement officers already delayed by two months given Covid 19 impact. • Lockdown likely to have improved effectiveness of community communication networks (including digital)	• Delays in recruitment would directly impact ability effectiveness of community engagement and so would decrease response in population sub-groups (related to some TAG groups). • Opportunity to benefit from improved communication networks

Statistical Design Phase 2 – Monitor/Counter

Area of Statistical Design	Potential Covid Implication	Potential Statistical Quality Impact
Response Chasing Algorithm	Assumptions used in Field Operations Simulation (FOS) maybe incorrect resulting in reduced effectiveness of Response Chasing Algorithm	Reduced overall response and increased variability.
Quality Assurance	• No impact on quality assurance of individual processes. • Availability or inaccuracy of administrative data would impact on our ability to assess variability in response data e.g. lower response in a population sub-group.	• Reliance would be on 2011 Census data and admin data that were available. Potential to miss poor response issues in some population groups which have recently changed
Business Intelligence/Management Information	• No impact (descoping field address check has had a minor positive impact on the amount of work needing to be covered	• None

Statistical Design Phase 3 – Process/Estimate

Area of Statistical Design	Potential Covid Implication	Potential Statistical Quality Impact
Data Cleaning	• No impact	• None
Census Coverage Survey	• No impact for short term disruption • Long-term disruption impacts on ability to recruit and train an effective field force • Inability to undertake a field operation would mean no address listing exercise.	• Long-term impact would undermine ability to adjust for coverage using standard design. • Without an address listing exercise no adjustment would be made for the missed addresses in the Census.
Under/Over Coverage Estimation	• Response to the main Census is impacted (as above)	• Increased variability would mean there was greater uncertainty in estimates for some areas/groups • Overall lower response would also mean greater uncertainty • Could be countered by planned statistical contingencies work using admin data (see below)
Quality Assurance	• Availability and quality of administrative (and survey) data which would be used to validate census estimates. • Data availability may also impact ability to make planned adjustments which are part of the standard design namely for number of households, babies and a national adjustment	• Ability to identify and explain inconsistencies between census and other data. • Ability to identify quickly and with confidence that there is a need to make further adjustments to the coverage strategy (invoking census contingencies) • ONS may not be able to publish and explain coherence between Census and other data – impacting user confidence. • Ability to make planned adjustments for households, babies and national adjustments directly impacts on quality
Further Development of Coverage Strategy (Contingencies)	• Availability and quality of administrative data to be able to make further adjustments beyond the standard design for any localised or national response issues (as experienced internationally)	• Ability to adapt the design to counter localised and national response accurately. • Impact would be greater uncertainty in the estimates overall resulting in a reduction in user confidence. • May disproportionately impact certain population groups or geographic areas.