APCP – Technical Minutes and papers

Meeting of 8 October 2021

TimeItemPresenter and PaperDescription
10:25Join call
10:30Introductions and
Mike HardieMake introductions if necessary.
Inform panel members of any
10:35New index number
methods in Consumer
Price Statistics
Kevin Fox & Peter Levell
ONS have developed a quality framework to assess index number methods against, to ensure any methods of calculating price indices are appropriate for the data source,
as part of introducing new alternative data sources in consumer price statistics. Through the Economic
Statistics Centre of Excellence, the ONS has commissioned a review of the framework, which is being led by Kevin Fox and Peter Level. Panel members are asked to feedback on the findings of the review.
11:35Rents DevelopmentNatalie Jones
Redeveloping private rental
market statistics
We present our proposed
method to measuring rental
price statistics. The aim of the
session is for the Panel to advise
ONS on the suitability of using its
preferred method for live
statistical production, and
provide any feedback only if
there are fundamental problems
with the proposed method.
11:55Weights 2022Chris Payne
APCP-T(21)15 Weights
update, 2022
This paper proposes an approach
for calculation of weights for the
Consumer Prices Index including
owner occupiers’ Housing costs
and the Consumer Prices Index
for 2022, taking into account
international guidelines, data
issues and the relevance of our
consumer prices weights
12:10HCIs Mortgage InterestDomenica Rasulo and Yasin
APCP-T(21)16 Measuring
mortgage interest
payments in the Household
Costs Indices
We present an update to our
research on the methodology for
mortgage interest payments. We
consider three approaches:
1. The current MIPs method used
in the RPI
2. Simple revaluation
3. Lenders' formula method
12:55AOB and date of next meetingMike HardiePublication status of presented
13:00Meeting close

Download agenda

Item 2 is a draft of a paper that will be published at a future date on the ESCoE website

Item 3 has not been published at this time, for more information please refer to the published minutes.

Mortgage interest payments

Weights update 2022

Meeting of 9 July 2021

TimeItemPresenter and PaperDescription
10:25Join call
10:30Introductions and
Grant FitznerMake introductions if necessary.
Inform panel members of any apologies.
10:40Update on Rental Indices DevelopmentChris JenkinsVerbal update
10.55RPI Revisions PolicyMike HardiePresentation of the proposed changes to the RPI Revisions Policy followed by discussion.
11:25Alternative data sources in the CPI and CPIH.Helen Sands
Integration of scanner and web-scraped data into consumer price statistics: aggregation and weights
Proposals and remaining challenges for the integration of scanner and web-scraped data into consumer price statistics, with a particular focus on aggregation of different data sources and their respective weights, as well as imputation for missing strata and consumption segments.
12:50Publication status of presented papers.Grant FitznerConfirmation of the publication status of the papers presented at the session.
12:55AOB and date of next meetingGrant FitznerNext meeting:
8th October 2021
13:00Meeting close

Download agenda

Meeting of 16 April 2021

TimeItemPresenter and PaperDescription
10:25Join call
10:30Introductions and
Grant FitznerMake introductions if necessary.
Inform panel members of any apologies.
10:40Progress update on private rental developmentMike HardieVerbal update
10.50Household Costs Indices: mortgage interest payments.Domenica Rasulo
APCP-T(21)07 Lenders’ formula method for mortgage interest repayments
APCP-T(21)08 Current mortgage interest payments methodology: a review
These papers present work to review the method used to calculate mortgage interest payments in the current Household Costs Indices estimates. The work focusses on two workstreams: the development of a new methodology which incorporates the lenders’ formulae used to derive households’ interest payments, and a sensitivity analysis of the parameters used in the current model.
11:25Web scraping: Product grouping methods.Hazel Martindale & Matt Eddolls
APCP-T(21)09 Dealing with product churn in web-scraped clothing data: product grouping methods
This paper discusses the methods and metrics under consideration for forming homogenous groups of products in web-scraped data, to reduce the effect of product churn. Two different methods of forming groups are presented along with discussion of how to assess the group properties. This paper
discusses the options currently under consideration and our plans for further work.
12:05Outlier detection and filtering methods for UK web-scraped and scanner dataLoes Charlton
This presentation reviews potential methods for anomaly detection to apply to scanner and web-scraped CPI data. Following a literature review, the suitability of short-listed methods is assessed by comparing their computation cost, the number and distribution of outliers they produce and their effect on price indices.
12:40Web scraping: Expenditure proxiesAlex Rose & Matt Love
APCP-T(21)10 Approximating Sales Quantities for Web Scraped UK Grocery Data
This paper assesses methods for producing sales quantity approximates for products at the elementary aggregate level. The considered methods focus on using product characteristics, such as price and page location, collected from web scraped data, along with item level sales quantities, provided by the retailer.
13:15Transparency: review of paper publication classificationsGrant FitznerA review of the current paper classifications used to determine publication status.
13:20Revisions policyMike HardieVerbal update
13:25AOB and date of next meetingGrant FitznerNext meeting:
9th July 2021
13:30Meeting close

Download agenda

Meeting of 15 January 2021

TimeItemPresenter and PaperDescription
10:25Join call
10:30Introductions and
Grant FitznerMake introductions if necessary.
Inform panel members of any apologies.
10:35Update on price collectionMike HardieTrends in unavailable items and the expected impact on local collection of the new lockdown.
10.45Rental Indices Development Plans and ProgressChris Jenkins / Natalie JonesAn overview of our plan and progress of developing our private rental price statistics.
11:10Weekly online price changes for food and drink items methodologyHelen Sands
APCP-T(21)04 Weekly online price changes for food and drink items methodology
Challenges presented by producing weekly price indices, as well as our plans for improving the indices in the coming months
11:35Research indices using scanner dataJo Corless
APCP-T(21)01 Research indices using scanner data
The latest research undertaken into the use of scanner data in our consumer price indices
12:00Refunds in scanner dataGeorge Bettsworth
APCP-T(21)02 Refunds in scanner data
Preliminary advice on how to consider refunds in scanner data
12:25Classification of Alternative Data SourcesLiam Greenhough
APCP-T(21)03 Classification of Alternative Data Sources
An update on clothing classification and research for grocery classification
12:50AOB and date of next meetingGrant FitznerNext meeting:
16 April 2021
13:00Meeting close

Download agenda

Download minutes

The January APCP-T meeting was held over two dates to allow further work and discussion of the item on rental indices development plans and progress. Items 3 and 4 were discussed on 15th January and 19th February, and the remaining items were discussed on 15th January.

Part 1:

  • 15th January 2021
  • Teleconference
  • 10:30 – 12:00

Part 2:

  • 19th February 2021
  • Teleconference
  • 13.00 – 14.00

Members in attendance

  • Mr Grant Fitzner (ONS, chair)
  • Mr Mike Hardie (ONS)
  • Mr John Astin
  • Mr Peter Levell
  • Dr Jens Mehrhoff
  • Prof. Paul Smith
  • Dr Martin Weale
  • Dr Antonio Chessa


  • Mr Huw Pierce (ONS)
  • Mr Chris Payne (ONS)


  • Ms Aimee North (ONS)
  • Mr Chris Jenkins (ONS)
  • Ms Natalie Jones (ONS)
  • Mr Liam Greenhough (ONS)
  • Ms Jo Corless (ONS)
  • Ms Helen Sands (ONS)
  • Mr George Bettsworth (ONS)


  • Ms Sofia Poni (ONS)
  • Ms Tanya Flower (ONS)


  • Dr Gareth Clews (Methodology, ONS)
  • Mr Rupert de Vincent-Humphreys
  • Prof. Bert Balk
  • Prof. Ian Crawford
  • Mr Jonathan Athow (ONS)
  1. Introduction and apologies
    • Mr Fitzner opened the meeting and passed on apologies from members unable to attend.
    • Two actions were carried forward from the previous meeting. Establishing whether rateable values were used in the CPIH historic series remains ongoing. This will feed into the second action carried forward: the methodological paper covering the data and modelling used for the CPIH historic series is expected to be published in Q1 2021.
  2. Update on price collection
    • Mr Hardie gave an update on changes to price collection due to the current level of COVID restrictions. Since November 2020, price collection reverted to online only with price collectors working from home. This will also be the case in the base period, January 2021, and will be reviewed on a monthly basis. The list of unavailable items is reviewed monthly and is referred to in the key points of the CPI publication.
    • An article explaining changes to price collection procedures and the considerations for calculating the 2021 weights, discussed in the December 2020 APCP-T meeting, is being drafted by Mr Payne for publication on 11th
  3. Rental Indices Development Plans and Progress (15th January)
    • Ms North presented on developments to private rental price statistics, covering proposed methodological changes and a review of preliminary results. The new methodology utilises Valuation Office Agency transaction level data allowing for more granular outputs by region, property type and bedroom category. A hedonic regression model is used to predict prices for rental properties based on a selection of property characteristics. Two forms of regression model are under consideration: a general linear model (GLM) with and without interaction terms, and a random forest.
    • Mr Fitzner highlighted that an update to the private rental prices development plan would be published on the 19th January 2021.
    • Dr Chessa noted from his own experience of hedonic modelling the importance of accounting for variable interactions and was pleased to see this was a consideration in the current work.
    • Dr Weale suggested that some of the anomalous results may be a result of overfitting the model and that removing some of the independent variables might improve performance. He enquired about the functional form of the model, as there were additional considerations for converting a log normal distribution back to a price. Finally, he queried the sample sizes at the most granular level of analysis, as certain local authorities may have very small numbers of certain property types. Ms North acknowledged this concern and described ongoing analysis to determine appropriate combinations of geography and property characteristics that would allow robust estimates.
    • Dr Mehrhoff highlighted the paradigm change involved in using a machine learning approach (i.e. random forest) for hedonic modelling, asking if it was sufficient for the model to make reliable predictions or if there was also a requirement for the model to be explainable in economic terms. He also highlighted that random forest models perform poorly with time dummies as the fitting process generally discards them, and that this can make the models unstable. He suggested using a shrinkage model (such as lasso) with explicit interactions as a compromise between the control of GLM and the flexibility of random forest. Mr Levell noted that random forest models have an inherent opacity that makes comparisons difficult.
    • Prof Smith queried the choice of 14 months as the rolling period for the dataset, when 10 months is the average contract length. Ms North explained that 10 months would be ideal, however there are two practical limitations. The first is that relatively few contracts are exactly 10 months long, the average being a balance between six or 12 months which are more common. The second is that data is collected by rent officers who are only prompted to visit properties annually. With a ten-month window this leads to far fewer data points and therefore a more volatile index.
    • Mr Levell highlighted that changes in the composition of the rental market in recent years makes the hedonic adjustment especially important. The observation that the random forest model gives a different index value to GLM may reveal something about how rental patterns at the local authority level are treated. Comparing GLM with other models that explicitly incorporate local authority interactions could illuminate this question.
    • Several panel members were interested in more details of the model fitting and asked if a technical paper was in preparation. Mr Jenkins confirmed that there is.

ACTION 1: Ms Jones to provide more detail on the model specifications

ACTION 2: Ms Jones to share a draft of the technical paper with Panel members

  1. Update on Private Rental Statistics Development (19th February)
    • Mr Fitzner emphasised the experimental nature of the results being presented and the rigorous quality assessment they will undergo before being introduced to the headline indices.
    • Ms North presented an update on progress since the APCP-T meeting on January 15th. Following feedback from that session ONS have tested four further models in addition to the two models presented in January. These are: weighted least squares (WLS) with the interactions that could be implemented and with and without shrinkage applied, WLS using longitude and latitude so that all interactions could be used, and a random forest model with constraints on tree size (“pruning”).
    • Three of these models have already been discarded by ONS. WLS using longitude and latitude with interactions modelled geographically adjacent local authorities in a similar way, which was felt to be unrealistic. The random forest model using longitude and latitude without pruning applied produced unacceptably volatile results. The WLS model using local authorities with the interactions that were able to be applied and shrinkage applied gave almost identical results to the same model without shrinkage, however it does not provide the statistical metrics (e.g.: R2, mean squared error) required to produce observation weights.
    • The three remaining models for consideration are: WLS with no interactions (hereafter “model 1”), a pruned random forest (“model 2”), and a WLS model with selected interaction terms (“model 3”). The model preferred by ONS is WLS with no interactions, as it is more transparent and easier to quality assure than a random forest and the results from WLS with interactions diverge markedly from the other two.
    • Mr Fitzner queried an anomaly in the chart of England annual percent growth. Ms North confirmed that the upward trend at the end of the WLS with interactions series (contrasting with the downward trend of the other two models) was most likely explained by the interactions in that model.
    • Mr de Vincent-Humphreys noted the consistency between models 1 and 2 and asked if this was reflected at smaller geographies and when comparing rental levels instead of index values. Ms North confirmed that all three models were in close agreement at lower geographies, with models 1 and 3 being almost identical. In respect of rental prices, models 1 and 3 were in close agreement while information on model 2 was not available at the meeting.

ACTION 3: Ms North to provide details of the price level estimates for all models at the regional level.

  • Dr Weale commented that the models predicted a geometric mean of rents (as expected from a logarithmic model) and queried if this was consistent with other ONS price statistics. Ms North confirmed it was consistent with the HPI, had been peer reviewed internally and is consistent with international best practice for housing market indices.
  • Crawford asked if confidence intervals could be provided for the models. Ms North confirmed that these were available for the WLS models. Prof Smith recommended bootstrapping as a way of establishing confidence intervals for the random forest model.

ACTION 4: Ms North to provide details of the confidence level estimates for the WLS models.

  • Prof Smith suggested using cross-validation and analysing prediction errors to give a more objective basis for recommending one model over the others. Prof Smith also raised concerns that the similarity between models 1 and 2 may arise because the pruning mechanism in model 2 has removed all the interactions. Mr Fitzner requested a sensitivity analysis be performed on the models to address these concerns.

ACTION 5: Ms North to perform the sensitivity analysis on the models and share the results with the panel.

  • Mr Levell asked for greater detail on which interactions were driving the differences between the two WLS models and proposed progressively adding interaction terms to understand the sources of difference. Mr Levell also suggested that controlling for the socioeconomic status of an area (via the ACORN variable) might suppress some of the modelled rent increase.

ACTION 6: Ms North to investigate the use of ACORN in relation to endogeneity

  • Mr de Vincent-Humphreys enquired about the likelihood of revising the back series of CPI and CPIH in light of these new methods, whether the methods were consistent with HPI in its use of hedonic regression, and for confirmation of the timeline for introducing these new methods into production. Mr Hardie confirmed that further analysis will be carried out during 2021, with a parallel run phase under consideration. No revisions will be made to historic CPI and CPIH values, consistent with ONS policy. Ms North described the parallels and divergences between the HPI and Rents Development processes.
  • Balk averred that the opacity of the random forest model was disqualifying, and without understanding the economic meaning of the interaction terms their use was unsafe. This leaves model 1 as the preferred option.
  • Dr Mehrhoff framed the decision as a balance between the better predictions of a random forest vs the transparency of the WLS model, but only if the random forest’s predictions really were better as measured by mean squared errors or R2. The preferred model (model 1) sits well with international recommendations and practices.
  1. Weekly online price changes for food and drink items methodology
    • Ms Sands described the process for compiling the weekly price indices that have been produced during the COVID-19 pandemic, highlighting that this is the first time alternative data sources have been used in a production situation. As many of the decisions taken in the development process were taken at short notice this was an opportunity to review these choices and consider how to take the indices forward. Areas of focus include product identification, outlier detection, and choice of index formula (currently GEKS-J with a five-week movement splice).
    • Dr Weale queried why the weekly index was using different samples of products to CPI. Ms Sands confirmed that this was partly due to the difficulties in product identification given that local collectors do not collect product IDs, and partly to leverage the greater coverage that the web dataset offers.
    • Dr Chessa complimented the work of ONS on alternative data sources and highlighted his own research on the topics discussed in the paper, offering to circulate a report on this to panel members. His analysis showed that indices with movement splices are prone to drift and that short windows can create problems with seasonal products. A potential solution could be to use an expanding window instead. Dr Chessa described another approach using a 25-month window with the lack of back data being addressed through imputation, however it was not appropriate for this situation. Dr Mehrhoff observed that 25-months was the emerging international consensus for treating seasonal products. There is a risk however, that if price trends within the basket diverge then the characteristicity of the index may be compromised.

ACTION 7: Dr. Chessa to provide a copy of his multilateral indices research report for circulation to panel members.

  • Ms Sands described the level of interest from other government departments in the weekly statistics. Mr Fitzner said it was likely that the weekly index will continue in some form after the COVID-19 pandemic due to this level of interest. Further analysis and stakeholder engagement are required to understand how the index can provide the most value in the longer term.
  • Dr Mehrhoff raised the opportunities that big data provided for identifying emerging trends and enquired about a comparison with conventional headline indices. If the weekly index is to become a permanent publication, then users will require greater detail on its methodology. Ms Sands offered to circulate comparisons with CPIH to panel members.

ACTION 8: Ms Sands to provide comparisons of the weekly index with CPIH.

  • Mr Astin praised the volume and quality of work by ONS on alternative data sources and enquired if these sources could lead to the end of in-person price collecting. Ms Sands averred that while meaningful volumes of expenditure continued to take place in independent shops there would be a requirement for local collection. Mr Hardie added that the RPI requires local collection, and therefore it would be required to continue until at least 2030. Beyond that it may be possible to refocus local collection efforts away from supermarkets and more towards independent stores, perhaps in support of regional price statistics.
  • Prof Smith expressed interest in the distributions of price levels, suggesting that if the distribution was not normal then using a multiple of standard deviation as an outlier detection method was unsuitable. He also questioned the implicit weighting towards stores with larger product ranges. While this might be appropriate for supermarkets vs independent retailers in groceries, in other markets smaller specialist retailers may have larger ranges than larger, more general outlets. Ms Sands confirmed that the price distributions were not normal and improved outlier detection methods were an ongoing topic of research. The choice of product range as a basis for weighting was made due to the difficulties encountered using market share directly. For the retailers being used currently analysis indicates that product range mirrors market share fairly well, however if the number of retailers expands this may need to be revisited.
  1. Research indices using scanner data
    • Ms Corless presented findings from research undertaken using scanner data. Retailers have provided data covering a range of goods, including groceries, homeware, clothing, electronics and fuel. Advice was sought from the panel on three topics:
      • Time coverage. Scanner data is provided by retailers aggregated by week; however, Consumer Price Statistics require prices to be attributable to a single month and some weeks span month boundaries. Proposed solutions are to exclude weeks that straddle months or only use the first three weeks of the month (to align with production timescales), but there is concern that this could introduce bias.
      • Product relaunches. Products are sometimes relaunched with different identification codes creating difficulty in maintaining continuity of data. An approach based on text matching product descriptions is under development, with an intention to test this using a dataset with linked product IDs. However, other NSIs apply alternative approaches such as expenditure monitoring to identify relaunched products.
      • Handling discounts: Discounts can take several forms, requiring a decision on which types to include and exclude from the index. Simple price reductions are straightforward to capture, but multibuy discounts and discriminatory discounts (e.g. loyalty cards) are more challenging. There is also a consideration on whether consistency is required between scanner data and local collection.
    • On time coverage, Prof Smith suggested constructing a weekly unit value index (UVI) and modelling values for those weeks that are split between months. The weekly index can then be averaged over a calendar month. Alternatively, it may be possible for index construction purposes to redefine a month as a collection of weekly periods. On handling discounts, Prof Smith expressed a preference for capturing as much information as possible, noting that significant proportions of expenditure are covered by loyalty card discounts. One way of addressing this is to treat items bought with and without a loyalty card as separate products.
    • Mr Astin asked if concerns over introducing bias by discarding weeks could be addressed by testing. Ms Corless agreed that it could. Dr Weale postulated that systematic bias would only arise from discarding weeks if retailers were engaging in behaviour that he found unlikely. While anomalies may occur in individual months, over the long term these were likely to cancel out.
    • Dr Chessa shared his experience of testing for bias when choosing which weeks to include. For supermarket and fuel data large improvements in accuracy were found when adding second and third weeks to the calculations. In the case of air fares adding a fourth week was necessary for months with moveable holidays (e.g. Easter).
    • Dr Mehrhoff offered to share the results of his own testing surrounding the time coverage question and encouraged ONS to carry out their own tests. He then described the difficulties that Eurostat encountered using text matching to identify product launches in the absence of other metadata. One approach to mitigating the effect of product relaunches is to use broader product categories. This introduces a risk of unit value bias but give the benefit of greater continuity. For discounts and particularly loyalty cards, Dr Mehrhoff advocated considering time consistency in the index when deciding what to include.

ACTION 9: Dr Mehrhoff to provide details of his scanner data research.

  1. Refunds in scanner data
    • Mr Bettsworth described research into the treatment of refunds in scanner data. Refunds are recorded differently in the datasets provided by different retailers; some log them as individual items while others include them in aggregate statistics. Other NSIs do not generally regard refunds as a problem, especially not for perishable items, although many had not fully considered the issue.
    • Where refunds are separately logged it is possible to perform analysis on their frequency, and more importantly remove them when obtaining price information. Refunds are far more common in some product categories (e.g.: clothing, jewellery) than others (e.g.: food). Where refunds are included in aggregate figures there are editing techniques that can be applied to extract price information. Current areas of investigation are whether refunds affect the final index and the consequences of choosing one editing method over another.
    • Mr Astin enquired about the treatment of vouchers, for example discounts offered above an overall spending threshold. Ms Sands confirmed that discounts that were applied to an entire basket rather than a single product are not usefully reported in the datasets. Dr Mehrhoff shared his experience that voucher transactions were recorded as positive quantities and negative expenditures and were likely to be removed by data cleaning processes. Responding to Prof. Smith, Mr Bettsworth confirmed that refunded quantities are reported as negative amounts in the datasets. Prof. Smith asked if this mirrors the treatment of refunds in the Living Costs and Food survey, and how refunds are treated in production currently.
    • Dr Mehrhoff outlined concerns about timing the processing of refunds. If a purchase is included in an index calculation in one month but refunded after publication, this is difficult to handle in an environment where indices are not revised. The problem is complicated further when using multilateral indices with splices, and there is no consensus on the appropriate way to handle this. Dr Mehrhoff suggested running case studies on various refund scenarios to understand the possible outcomes, and Dr Weale concurred. Mr Bettsworth highlighted the difficulties of linking refunds to the original sale, noting that other NSIs had also encountered this problem. Mr Levell suggested using information from the retailers that provide disaggregated refund data to model the volume of refunds across all datasets.
    • Dr Chessa described findings from his own research, confirming that some categories experience a high enough volume of refunds to distort prices. He suggested that if price lists were available it may be possible to build a model that would allow adjustments to be made to the dataset to compensate for refunds.
  2. Classification of Alternative Data Sources
    • Mr Greenhough gave a presentation summarising progress on classifying clothing and groceries in scanner data and outlining further plans for development.
    • Good progress has been made applying a machine learning approach to clothing, with focus turning to achieving performance gains, understanding the level of bias due to misclassification and preparing the pipeline for production use.
    • A machine learning approach is thought to be less suitable for groceries however, as products are more likely to have explicit identifiers that can be used instead. Moreover, smaller training datasets and the more granular COICOP structure for groceries increases the risk of misclassification, and the consequences of misclassification for the overall index are more severe for some products due to the higher weight that some groceries carry. Mr Greenhough proposed potentially using machine learning as a method of machine-assistance for improving the efficiency of manual classification, with products still receiving manual scrutiny as assurance. Also potentially exploring whether very low-expenditure products can be excluded without impacting elementary aggregates.
    • Dr Mehrhoff asked if further information could be provided on the precision/recall performance of the classifiers. A scatter plot of product category weight and precision/recall would indicate if the most important categories for the index were achieving sufficient predictive success. Mr Greenhough confirmed these could be made available.

ACTION 10: Mr Greenhough to provide precision/recall scatter plots.

  • Dr Mehrhoff noted from his experience firstly that manually reviewing higher weight categories is often valuable, and secondly that classification performance is usually more dependent on the quality of input data than on the choice of classification algorithm.
  • Prof Smith expressed concern over the risk of propagating incorrect classifications and suggested carrying out further research into the topic.
  1. AOB and date of next meeting


No.ActionPerson Responsible
1Provide more details on the private rentals model specification.Ms Jones
2Provide a draft of the private rentals technical paper for circulation to panel members.Ms Jones
3Provide details of the price level estimates for all models at the regional level.Ms North
4Provide details of the confidence level estimates for the WLS models.Ms North
5Perform the sensitivity analysis on the models and share the results with the panel.Ms North
6Investigate the use of ACORN in relation to endogeneityMs North
7Provide a copy Dr Chessa’s multilateral indices research report for circulation to panel members.Dr Chessa
8Provide comparisons of the weekly index with CPIH.Ms Sands
9Provide details of Dr Mehrhoff’s scanner data research.Dr Mehrhoff
10Provide precision/recall scatter plots for the product classifiers.Mr Greenhough

Meeting of 18 December 2020

TimeItemPresenter and PaperDescription 
10:25Join call
10:30Introductions and
Mike HardieMake introductions if necessary.
Inform panel members of any apologies.
10:35Approach to Consumer Price Indices Weights 2021Chris PaynePresentation on the proposed approach to calculating CPI and CPIH weights for 2021
10:50Round table discussionPanel membersComments and questions raised by the presentation.
12:00Meeting close

Download agenda

18th December 2020 – Teleconference -11:00 – 12:00

Members in attendance

  • Mr John Astin
  • Mr Mike Hardie (ONS, chair)
  • Mr Peter Levell
  • Dr Jens Mehrhoff
  • Prof. Paul Smith
  • Dr Martin Weale
  • Dr Gareth Clews (Methodology, ONS)
  • Prof. Ian Crawford
  • Mr Rupert de Vincent-Humphreys


  • Mr Huw Pierce (ONS)
  • Mr Chris Payne (ONS)


  • Dr Antonio Chessa
  • Prof. Bert Balk
  • Mr Grant Fitzner (ONS)

1. Introduction and apologies

  • 1.1 Mr Hardie welcomed attendees to the Advisory Panel on Consumer Prices-Technical (APCPT) meeting and introduced Mr Pierce who is assuming the secretary role going forward.
  • 1.2. As an interim meeting there were no previous actions to discuss.

2. Approach to Consumer Price Indices Weights 2021

  • 2.1. Mr Payne gave a presentation on the proposed approach to calculating Consumer Prices Index (CPI) and Consumer Prices Index including Owner Occupiers’ Housing costs (CPIH) weights for 2021 and summary analysis of the calculated weights. Panel members were invited to comment on 1) the process for arriving at the weights and 2) the indicative weights themselves.
  • 2.2. The proposed approach adopted for Retail Prices Index (RPI) weights in 2021 is to use Living
    Costs and Food Survey (LCF) data for the period from July 2019 to June 2020. For CPI and CPIH the proposed approach is to align with international statistical guidance, along the lines set out in recent Eurostat guidance. This has the benefit of retaining international comparability and best practice.
  • 2.3. Adjustments will be limited to categories that are most heavily affected and/or show sustained changes in spending.
  • 2.4. Estimates of the new weights were presented, acknowledging some limitations.
  • 2.5. The new 2021 weights were compared with weights calculated under the conventional process (using 2019 spending data instead of 2020) and the weights used in 2020. While most classes experienced relatively small changes due to the altered process, there were some that experienced larger movements.

3. Round-table discussion

  • 3.1. Panel members were invited in turn to offer feedback and raise questions on the presentation.
  • 3.2. Dr. Mehrhoff commented on the general uncertainty surrounding 2021 and how reflective our estimates would be (as a result of Eurostat guidelines) should circumstances change. Weights should account for 2020, but also anticipate behaviour in 2021 dependent on how far the economy reverts to normal. Dr. Mehrhoff and Prof. Crawford were supportive of the idea of only updating categories that showed a sustained change in consumer behaviour.
  • 3.3. In response to questions from Mr Astin, Mr Payne confirmed that the procedure for updating the RPI weights uses LCF from July 2019 to June 2020 and therefore follows a different schedule to CPI and CPIH, such that the impact of changes to spending patterns is less and therefore there will be no changes to the RPI process. Mr Astin also asked about what the ideal index would be in these circumstances. Conventional approaches to building price indices have little to say about the current situation we face and therefore it is difficult to make a case for any particular index to be the ideal one for benchmarking.
  • 3.4. Mr Hardie emphasised the value users place on continuity across time and international comparability in price indices. For these reasons it made sense to follow Eurostat guidance. Mr. de Vincent Humphreys raised the importance of how we explain our approach to users and the public.
  • 3.5. Prof. Smith and Mr Levell both raised the topic of smoothing the weights, acknowledging the complexity involved in doing this. Mr Levell asked if there was a statistical test being applied as to what constituted a sustained change in spending behaviour, as this would act as a threshold for identifying weights that need adjustment. Mr Payne referred to work done by the Australian Bureau of Statistics on this point and suggested that ONS may follow a similar approach to them.
  • 3.6. Dr. Weale suggested thinking about what price change we wanted to show and using that to guide the approach. If the change between 2021 and 2020 is the key comparison then the approach is probably the right one.
  • 3.7. Panel members queried the process for creating weights for 2022 and the “exit strategy” for this interim process, i.e. how we return to normal procedures after the pandemic.
  • 3.8. Following a comment on the European situation, a panel member asked if the USA or Canada had declared their approach to calculating 2021 weights. Mr Payne confirmed he would be discussing with both countries.
    • Action: Mr Payne to report back on the approach taken by the USA and Canada

  • 3.9. Panel members gave broad approval of the approach taken by ONS and recognised the  complex nature of the problem at hand.


  1. Update the group on the approach taken by the USA and Canada – Mr Payne
  2. Provide the presentation slide pack to Bert Balk Mr Payne

Download minutes

Meeting of 9 October 2020



Draft consumer prices development plan 2021*

CPIH: Producing a historical series for the period 1950 to 1987*

On Household Costs Indices*

*This paper has not been published at this time, for more information please refer to the published minutes.

Meeting of 10 July 2020



Meeting of 17 April 2020



Contingency plan for ongoing price collection and compilation during the Covid-19 pandemic*

CPIH: Producing a historical series for the period 1947 to 1987**

Progress on automated classification of web-scraped clothing data**

Estimation and presentation of information on the accuracy of the Consumer Price Index**

*This is a draft of an article that has now been published and is available on the ONS website.

**This paper has not been published at this time, for more information please refer to the published minutes.

Meeting of 17 January 2020



Terms of reference*

Refining the higher education component of the Household Costs Indices

Expanding the measurement of interest payments on financial debt**

The winning formula? A framework for choosing an appropriate index method for use on web scraped and scanner data

* This paper is a draft update of the Terms of Reference, which has now been published on the UK Statistics Authority website.

** This paper is work in progress and has not been published at this time

Meeting of 8 November 2019



Proposed text for the consultation on the future of the Retail Prices Index*

* This paper is a partial draft of the consultation which has now been published on Citizen Space

Meeting of 6 September 2019



Alternative methods and data sources for package holidays – update*

Price collection in a no-deal scenario*

Updating the Data Source for Shop Type Weights*

Collecting Discounted Prices

Discussion on unweighted index number formulae**

* This paper has not been published at this time, for more information please refer to the published minutes.

** This paper is a draft and has not been published at this time.

Meeting of 17 May 2019



The conceptual foundations of the Household Costs Indices (ESCOE workshop)

The conceptual foundations of the Household Costs Indices: Annex A

Alternative methods and data sources for package holidays*

Shortlisting Appropriate Index Methods

Classification metrics guidelines

* This paper has not been published at this time, for more information please refer to the published minutes.

Meeting of 11 January 2019



The use of unweighted indices in ONS’s consumer price statistics

Assessment of small area estimation for regional CPI expenditure weights*

The use of approximate expenditure weights for web scraped data

Quality adjustment review

Note from Jens Mehrhoff in response to APCP-T(19)01: The use of unweighted indices in ONS’ consumer price statistics

* This paper is a draft of an article that has now been published and is available on the ONS website.

Meeting of 14 December 2018



Calculating a price index for capital mortgage repayments*

Calculating a price index for student loan repayments*

* This paper has not been published at this time, for more information please refer to the published minutes.

Meeting of 7 September 2018



Updating the data source for shop-type weights*

Variance estimates for the Consumer Prices Index including owner occupiers’ housing costs*

Proposed pipeline for processing alternative data sources

The use approximate expenditure weights for web scraped data in consumer price indices*

* Update 04/02/19: Minutes updated to correct a minor error.

Meeting of 11 May 2018



Calculating a price index for student loan repayments*

Draft work programme for consumer price statistics*

Investigating the use of web scraped data to improve clothing measurement*

Extending the classification structure

* This paper has not been published at this time, for more information please refer to the published minutes.

Meeting of 15 January 2018



Producing standard errors for CPIH*

Feasibility study into producing CPIH-consistent inflation rates for UK regions

Producing a historical series for CPIH*

Measuring changes in used car prices

* This paper has not been published at this time, for more information please refer to the published minutes.

Meeting of 15 September 2017


Minutes (updated February 2018)

A theoretical framework for the Household Costs Indices*

Review of quality adjustment in consumer price statistics

Assessing the feasibility of web scraped data within current collection methods

Smoothing volatile weights

AOB – What is the likely scale of the effect of using the current chain linking algorithm

* This paper has not been published at this time, for more information please refer to the published minutes.

Meeting of 19 May 2017



The measurement of real household incomes in the UK: Options for a matched approach*

Including the cost of finance in CPIH and HCIs

Inflation for household groups: calculation of weights

Consumer prices work programme*

AOB – Chain linking algorithm question

* This paper has not been published at this time, for more information please refer to the published minutes.

Meeting of 20 January 2017



Terms of Reference

RPI and CPI: a tale of two formulae

Re-addressing the formula effect

Review of web-scraped price indices

Additional correspondence – OOH weight for imputed rentals

Meeting of 1 September 2016


Minutes – Amended 14 February 2017

Inclusion of Council Tax in CPIH

Meeting of 10 May 2016



Written Submission

Meeting of 22 January 2016


Draft Terms of Reference


Consumer Price Indices in the UK

Dr Courtney’s notes on the minutes