APCP-T(25)03 UK House Price Index (UK HPI) monthly imputation improvement

Advisory Panel on Consumer Prices Technical Panel 11 April 2025 presentation slides

Members in attendance

• Mr Stephen Burgess (ONS, Chair)
• Mr Grant Fitzner (ONS)
• Professor Paul Smith
• Dr Jens Mehrhoff
• Mr Peter Levell
• Mr Rupert de Vincent-Humphreys

Secretariat

• Ms Aimee North (ONS)
• Ms Rifah Abdullah (ONS)

ONS attendees

• Mr Chris Payne (ONS)
• Ms Abi Casey (ONS)
• Mr Malik Khalid (ONS)
• Dr Mario Spina (ONS)
• Ms Emily Hopson (ONS)
• Mr Andrew Carey (ONS)
• Mr Peter Bailey (ONS)
• Mr Liam Greenhough (ONS)
• Mr Markus Sova (ONS)

Apologies

• Professor Rebecca Killick
• Professor Ian Crawford
• Dr Martin Weale
• Mr Mike Hardie (ONS)

1. Introduction and apologies

1. Mr Burgess opened the meeting and passed on apologies from members unable to attend.

2. Proposed method improvement: HPI monthly imputation 

1. Ms North introduced a paper on monthly imputation methods for house prices, following on from an earlier discussion at the January 2025 panel.
2. Mr Khalid summarised the content of January 2025’s APCP-T(25)01 HPI monthly imputation paper, explaining that the proposed method improvement to HPI’s monthly imputation for Great Britain aims to address the observed over-estimation in early provisional HPI estimates.
3. Mr Khalid presented impact analysis of several imputation methods which were compared against the UK HPI’s current outputs, including assessment of the effectiveness of the temporary ‘new build pooling’ method relative to the effectiveness of improved imputation methods.
4. Mr Khalid summarised that:
   1. ‘New build pooling’ was shown to be much less effective at reducing HPI revisions than any of the four imputation options presented in this paper
   2. All imputation options significantly reduced HPI revision size and all performed similarly well.
   3. ONS proposed to discontinue the temporary ‘new build pooling’ approach and simultaneously implement K nearest neighbour (KNN) monthly imputation. This was found to be more effective at reducing overestimation of new build house prices compared with the current approach, performed similarly well as the other imputation options assessed, was computationally efficient and suitable for a monthly production timeline, and was more similar to HPI’s existing annual imputation methodology than median imputation.
5. A panel member queried why plotting the price trend of the initial (1^st) estimate has a different pattern to the trend of the latest estimate (shown by Figure 1 in the APCP-T(25)03 HPI imputation paper). Mr Khalid explained that this was due to the lag in transactions and attributes data: that the initial data sent to ONS for calculation of provisional HPI estimates does not contain all transactions for that period and does not contain all attributes information (particularly for new builds) since it takes time for transactions to be processed and records created or updated to be available for inclusion in the following month’s UK HPI estimates. This lag in data availability necessitates revisions in the UK HPI over several months and leads to differences between the trend plotted using just the initial (1^st) estimate for each period and the trend plotted using the latest revised estimate for each period.
6. Panel members agreed that all imputation options demonstrated improved calculation of HPI estimates compared to the current method and commented that all four presented imputation options appeared to produce similar results to each other, with similar impact on revisions.
7. A panel member commented that while the proposed improvement reduces over-estimation in provisional price estimates, none of the imputation options entirely eliminated over-estimation in provisional price estimates. They asked why implementation of this improvement is a priority for ONS when it only partially addresses the revision issue in UK HPI.
8. Ms North explained that initial price over-estimation could not be entirely eliminated by this imputation improvement because over-estimation is also driven by the inherent pattern in transactions being reported to HM Land Registry. However, this improvement addresses a source of over-estimation bias in the current HPI methodology, and ONS is keen to take action to improve accuracy of provisional estimates. ONS’ two-step plan previously presented to APCP-T in January 2025 outlined ONS’ intention to implement this proposed improvement as an interim solution in Stage 1, and to follow up later with a full methodology review in Stage 2, which will include reviewing the monthly imputation method to assess for potential realisation of further improvements.
9. Another panel member asked for more detail about the rate of missingness for these key variables in the property attributes data source to understand what proportion of records imputation would need to be applied for. ONS took an action to follow up with more detail on the missingness rate.
10. The panel member then queried whether there was a risk that imputation itself could introduce a bias. Ms North clarified that analysis demonstrated that imputation would significantly improve the new build regression coefficient, and that as the true attributes data for a given property becomes available (and missingness rate decreases) in subsequent months, the vast majority of imputed data are replaced with true data, reducing the scale of imputation in final HPI estimates. Analysis also showed that all imputation options performed similarly, producing statistically similar price estimates, suggesting that KNN imputation did not cause a particular bias relative to the other options.
11. A panel member asked for validation analysis to be shared.
12. A panel member agreed that the KNN imputation methodology has value. However, they suggested that price should be used to impute property characteristics in addition to other attributes. Panel members raised several suggestions of further potential methodology improvements, to monthly imputation and other elements of HPI methodology for ONS to consider.
13. Mr Burgess thanked the panel members and took an action for ONS to provide the agreed additional information and responses to outstanding questions via follow-up correspondence.
14. Following receipt of the follow-up response, panel members agreed via correspondence that the proposed monthly imputation improvement was an improvement on the current methodology and expressed their desire for a full UK HPI methods review (Stage 2 of ONS’ two-step plan from January 2025). No panel members dissented to ONS’ plan to implement the proposed KNN imputation improvement and simultaneous discontinuation of the temporary ‘new build pooling’ approach as Stage 1’s interim improvement.
15. Following this, ONS’ proposed improvement and APCP-T’s feedback were discussed by the UK HPI Working Group on 28 May 2025. Members unanimously agreed to implement ONS’ proposed improvement to HPI’s monthly imputation for Great Britain and to simultaneously discontinue ‘new build pooling’. The date of implementation will be announced to users via the UK House Price Index reports on GOV.UK and via ONS’ Private Rent and House Prices, UK bulletin, in line with usual practice.

3. Retailer Stratification

1. Dr Spina presented an update on ongoing retailer stratification work investigating three potential approaches, summarising findings and presenting ONS’ recommendation to stop pursuing ‘no retailer stratification’ weighting. The three approaches were:
   1. multiple/independent, the current retailer stratification which uses ABS 3-year lagged data and is based on number of physical shops in the UK
   2. big/small, the new proposed method which uses ABS 3-year lagged data and looks at market share of retailer to determine its weight
   3. no retailer stratification (i.e. implicit weighting which reflects the frequency of the retailer in the data).
2. Dr Spina explained that the research investigated whether implicit weight calculations (c above) would be representative of retailer weights when incorporating the Alternative Data Source (ADS) retailers and presented two case studies, on scanner groceries data and clothing and footwear, used to assess this. The results of the analysis show that the implicit weights are not representative of retailer type weights.
3. Dr Spina further explained that the implicit weights derived from non-ADS retailers are from more recent periods than the ADS scanner data calculated from ABS, which are on a 3-year lag. Proceeding with implicit weighting once scanner data is implemented, would mean that weights would be calculated using source data from 2 different time periods because of the 3-year lag on the ABS data, which is not best practice.
4. Dr Spina highlighted that consumer spending habits show shift to online-only retailers and that this is not captured in the multiple/independent stratification weighting, and adjustments are needed.
5. Dr Spina explained that ONS’ recommendation is to not explore implicit weighting further, and focus on assessing the impact of big/small retailer stratification to present to the panel in July.
6. A panel member queried what the weights would look like at a granular level with explicit weighting. Dr Spina explained that similarly to the current approach, aggregation would be bottom up, only you would calculate market share by item. There was broad agreement that it would be better to not use implicit weights, although a panel member highlighted that it is rarely the weights which are the issue in consumer price index calculations but the prices themselves.
7. Another panel member raised that the types of stratifications such as region and retailer were decided a long time ago and suggested having a top-down look to decide if this stratification is the most appropriate way to address weighting.
8. Mr Payne highlighted the practical benefits of using big/small stratification such as reducing the annual reclassification workload. Mr Payne suggested sending around additional information on the implicit weights calculation to the panel via correspondence.

4. Grocery Scanner Data Impact Analysis

1. Ms Hopson presented the final impact analysis of grocery scanner data and updated the panel on the publication plan for the following year. She highlighted that the introduction of grocery scanner data would provide improved product coverage, allow analysis of promotions and capture 3 weeks of data instead of 1 or 2 days in a month.
2. Ms Hopson reiterated the planned methodology improvements and explained the reasons for differences between grocery scanner data and local collection series. She explained that the largest differences seen were due to the ad-hoc adjustments made during the COVID-19 pandemic that were not able to be replicated. Ms Hopson then presented some of the findings that would be discussed in the impact analysis such as the movements in alcoholic beverages and tobacco or cheddar cheese and the capture of branded products.
3. A panel member queried whether loyalty card discounts would be captured in the scanner data. Ms Hopson explained the loyalty card prices would indeed be captured but cannot be isolated in the dataset, therefore another possible reason for differences between the baseline and scanner data index.
4. Another panel member asked if plans were to implement this improvement in March 2026, which Ms Hopson confirmed. The panel member asked if the analysis would be extended to cover more recent time periods and the impact analysis continued ahead of implementation. They requested that more recent analysis be presented to the panel to assess if trends have continued in later time periods. Ms Hopson and Ms Casey confirmed the plan is to implement this in March 2026, that ONS is doing a parallel run and that further impact analysis will be done in live parallel run from May.
5. Mr Burgess commented that published impact analysis timeseries would be restricted to the first half of 2024, to reduce the risk of confusion around the latest published inflation estimates for months in 2025.
6. Another Panel member asked how this would impact Household Cost Indices (HCIs). Ms Casey clarified that this data will flow into the HCIs. Panel members agreed that the data should flow into the HCIs as well.
7. Panel members emphasised that care would be needed when publishing the data going forward to ensure there is no misinterpretation of its meaning.

5. PPI Methodology Correction

1. Mr Carey summarised the methodology challenge in the Producer Price Index (PPI) and that the issue relates to how the new base period is introduced during annual update to sales and the factor applied when price updating, leading to an error during chain-linking.
2. Mr Carey explained that this has resulted in a noticeable impact on headline PPI inflation measures in some time periods, so the publication has ceased until a correction is ready. Mr Carey explained the new methodology would require update to the base period every time the weights are updated. Mr Burgess highlighted that the impact affected areas of national accounts and UK trade.
3. A panel member raised that it is important to be clear regarding the terminology around base period to ensure there is no misunderstanding. Ms North agreed and stated they had a similar issue in HPI when discussing re-referencing.
4. A panel member queried exactly what type of indexation method was being used. Mr Sova explained a Laspeyres index calculation is planned; however, limitations in the availability of the source data for the base year means that the latest data is price uprated as an estimation of what it would be in the base year.
5. A panel member also asked if there were plans for additional potential development work beyond the scope of the current identified error in PPI. Mr Sova clarified that they were following international best practice and other ONS statistics use the same approach. He also raised that there was a need to correct the error and resume publishing PPI, so a balance would need to be achieved and perhaps a further review in the longer term after the correction would be appropriate to address some of the suggestions raised by the panel.
6. A panel member raised that it would be appreciated to be able to see an update and the full impact analysis, especially on the impact of services PPI.
7. Another panel member requested if a worked example can be sent around to provide the panel more clarity. Mr Burgess suggested that the worked example can be sent around after the meeting and for the team to come back for the July panel to provide an update.

6. Membership and Future Panel Members

1. Mr Burgess thanked the panel for all of their contributions throughout the years and discussed the need for transparency in appointments. He invited suggestions for future panel members and emphasized the importance of maintaining expertise and balancing skills.

7. Publication status of papers

1. It was agreed that the APCP-T(25)03 HPI imputation paper and slides would be published alongside the minutes.
2. It was agreed that the retailer type stratification slides would be published alongside the minutes.
3. It was agreed that grocery scanner data impact analysis slides would have market sensitive slides redacted and the rest published alongside the minutes.
4. It was agreed that the APCP-T(25)04 PPI methodology paper would not be published alongside the April 2025 minutes while active investigation was ongoing, but its publication status would be reviewed at the July 2025 APCP-T meeting, at which ONS plans to provide an update. Market sensitive slides will be redacted.

8. Any other business and date of next meeting

1. Seasonal Adjustment for CPI
   1. Mr Burgess provided an update on the seasonal adjustment for CPI, mentioning the handover of code and documentation and the ONS’s provisional intention to publish seasonally adjusted CPI in March 2026. He explained some time would be needed to arrange an appropriate way to productionise the process and the need for further consultation with stakeholders.
   2. Panel members raised concerns about the treatment of annual duty announcements and suggested thought needed to be given to what the user demand is in order to choose an appropriate way to manage it.
2. The next APCP-T meeting is scheduled for 18 July 2025.