Annex D: Quality adjustment of education

To generate quality adjusted attainment estimates for Education, the Office for National Statistics (ONS) takes account of educational performance, in terms of qualifications and grades achieved by students. This annex will discuss in greater depth two topics. Firstly, the wider UK National Accounts implications of this and secondly, how the Review has improved the method of applying qualification data to different cohorts.

D.1 UK National Accounts implications

It should be noted that the methods proposed in this Review look to align to UK National Accounts conventions, as per the Atkinson Report. The area this is most immediately pertinent is in relation to human capital.

Human capital is recognised in the national accounts through the spending on Education and Healthcare which creates it, and the wages which result from it. However, human capital itself is not recognised as a capital and the flows which relate to it are not considered as investment or earnings from capital: they are considered recurrent spending and compensation of employees. As described in Canry (2019), Dunn (2022) and Mubarak, Heys, Meirinhos, and Taylor (2024), this constrains efforts to incorporate human capital into national accounts and hence into this Review.

If human capital was recognised, the return to Healthcare and Education spending could be ‘monetised’ through a rate of return calculation treating the discounted sum of additional future earnings as the value created by this expenditure. This would have two effects: it would significantly increase the value added of these services, but it would also require substantial parts of this expenditure to be reclassified as Gross Capital Formation by the government sector.

Alongside this issue, there is a further issue related to a core Atkinson principle, around attribution. Lifetime earnings can go up with improved qualifications because of either improvement in skill levels as a result of gaining the qualifications, or because of a signalling effect where gaining qualifications reveals the individual as having higher innate ability. Because the latter cannot be attributed to government spending, there is an argument this metric should not be used to quality adjust public services. In addition, extra qualifications may be gained because of privately-funded education or private spending on tutors, materials, books or other educational factors. Again, this mitigates against using human capital as a quality adjustment.

Further, human capital will not be addressed in the System of National Accounts (SNA) 2025 for technical reasons and a broad agreement that insufficient international research has gone into developing the necessary methods. Mubarak, Heys, Meirinhos, and Taylor (2024) attempts to close this gap but this Review encourages national statistics institutes around the world to undertake the necessary research, so the opportunity offered by future SNA revisions is not missed again. Nevertheless, developments at the ONS around its new Inclusive Income and Wealth Accounts, which deliberately extend the asset boundary to include human capital, provide a potential mechanism to showcase the value of improving the understanding of human capital formation.

Finally, it is important to note that whilst human capital is the largest example of this problem, the same issues arise for spending on the environment as natural capital, spending on museums, galleries and national monuments as heritage capital, and defence, justice and other mechanisms to ensure the rule of law and defence of property rights, which could be conceptualised as social capital. In fact, large swathes of government spending could be reconceptualised as types of capital expenditure in this fashion. Obviously, such a sweeping change would call for wholesale revision of the methods proposed in this Review and currently used. Such a piece of work is outside the scope of this Review, but it is important to be aware of the alternatives and their potential impacts.

D.2 Applying quality adjustments

Attainment

The ONS does not directly apply growth rates in exam scores in the year they are achieved as this would imply that GCSE exam achievement is solely attributable to performance during year 11. Instead, the model considers education as a cumulative experience, where each year of education builds on existing knowledge and skills, and contributes in aggregate towards the final academic achievement.

For example, GCSE attainment data published for the academic year ending 2018 reflects the effectiveness of the teaching from Reception year (in this case, academic year ending 2007) to Year 11 (academic year ending 2018). As such, the “cohort split” approach applies weighted contributions from the new attainment data back to previous years.

The coronavirus (COVID-19) pandemic compelled reconsideration of this model because it is methodologically incorrect to attribute the effect of the pandemic into the back series. For example, if attainment dropped by 10% during the pandemic (academic year ending 2021), there is no justification for reducing the attainment in academic year ending 2018 for students who were in school at that time. The coronavirus pandemic did not affect previous time periods: if the coronavirus pandemic had not occurred the Review would have expected students to have otherwise demonstrated a similar performance to their immediate predecessors, that is the cohort who completed their examinations in, for example, academic year ending 2019.

Therefore, a key aim for Education during the second year of the public service productivity (PSP) Review was to revise and re-develop the cohort-split model to robustly account for the coronavirus pandemic’s influence on attainment. This development can be broken into two elements: the first is sourcing evidence to scale the impact of the coronavirus pandemic, and the second is how to augment the methodology to cope with this shock.

In the Public service productivity annual publication of April 2023, prior to the beginning of the Review, the ONS used learning loss’ metrics published by the Department for Education (DfE) and Education Policy Institute (EPI) to inform attainment from academic year ending 2020. These measures only applied to England schools, however in the absence of similar metrics for the devolved governments, the English learning loss data were applied to all school phases across the UK.

The learning loss measures represented the average months of learning lost in English and Mathematics for Primary and Secondary schools during the academic year ending 2020 and 2021, in comparison with pre-pandemic performance. Although these data served as appropriate proxy indicators of attainment, upon reflection and discussion with experts on this topic, the Review has concluded that this measure is not the best for its purpose.

Learning loss is not strictly a measure of academic performance, it is used to gauge how in front or behind one cohort is performing in comparison to another. This renders the measure inconsistent to what would typically be used in the model.

Learning loss potentially double-counted movements that were being accounted for in the quantity output model, which was designed to account for lost teaching hours during the pandemic. The quantity output model was modified to account for the impacts of challenges that arose during the pandemic such as remote learning and increased absence rates.

Learning loss was used to adjust the raw attainment index in academic year ending 2020, but it had no further impact on the subsequent ‘cohort-split’ model. Nevertheless, a break in the cohort model had to be applied at academic year ending 2019 as a result so previous years were not impacted by the loss in academic year ending 2020. The learning loss impact applied in academic year ending 2020 meant this year was ‘fixed’ and hence caused a break in the model whereby attainment in future years could not be ‘cast-back’ into academic year ending 2020. This led to a break in the calculations at this point.

The study commissioned by the DfE and EPI did not have data for academic years ending 2021 onwards. Although alternative studies are available which have assessed learning loss during the pandemic, the points raised in this annex emphasises the need for actual attainment data to inform quality adjustments for Education.

Following conversations with DfE and the Office of Qualifications and Examinations Regulation (Ofqual), the National Reference Test (NRT) was identified as an alternative indicator of academic performance. The NRT, commissioned by Ofqual in 2017, was implemented to monitor GCSE-level performance over time, and is a short English and maths exam which is delivered to Year 11 students in England who are due to take their GCSE’s that academic year. The questions reflect the kinds of questions that will be taken in a formal GCSE examination. A nationally representative 300 of the 3,452 secondary schools are selected to take the test.

The outcomes of the first NRT in 2017 were benchmarked against GCSE results in 2017 to establish a baseline for following years. In 2017, the test booklets were marked, and the proficiency of the students were obtained; the proficiency estimates were benchmarked against the proportions of students achieving grade 7 and above, grade 5 and above, and grade 4 and above at GCSE. From 2018, the proportions of students at these proficiency levels are compared to the 2017 baseline, which allows for changes in academic performance over time to be observed.

The NRT has key advantages in informing attainment over learning loss:

• The NRT is a direct indicator of GCSE-level performance and has a strong link to academic achievement that doesn’t apply to the learning loss measures.
• The NRT is an appropriate metric for the cohort model, as the years prior to the pandemic will have had some bearing on NRT outcomes, and it is consistent with data typically used in the model. This means that the cumulative nature of education can continue to be accounted for in ongoing PSP estimates for Education.
• The NRT is independent of teacher assessed grades and avoids debates around whether grade improvements were a result of better education or grade inflation.
• Data for the NRT are available in a consistently measured time series back to 2017, and it covers the academic years affected by the pandemic.

On the other hand, there are some limitations that do need to be acknowledged when using the NRT, such as:

• The NRT is only undertaken in England, but no similar measures have been uncovered for the devolved governments.
• The NRT only assesses performance in Maths and English, but not other subjects. This is unlike, for example, the Attainment 8 score (which was typically used before the pandemic) which considers a pupil’s average grade across 8 subjects.
• Although the NRT is an indicator of GCSE-level performance, it is not a formally recognised GCSE, therefore it is not entirely consistent with previous measures used to inform quality adjustment for Education.

Upon consideration of the strengths and limitations, the NRT is the only viable source of secondary school attainment during the pandemic. Therefore, for 2020 and due to the absence of alternative data, the NRT was also used to inform attainment for England and the devolved governments. GCSE attainment data has returned to pre-pandemic arrangements in academic year 2023 to 2024.

In terms of how the cohort-split model needed development to absorb this data, this falls into two parts: the first is how to cope with structural breaks around the coronavirus pandemic to cope with adjustments made only during the pandemic years, and secondly this work provoked a deeper review of the weights applied in the model to allocate attainment across all the years each cohort is in education.

To adapt the Cohort model in the years of the coronavirus pandemic, the Review had to add some methodological adjustments. The key elements, discussed in the following section were:

• How to address the years prior to the coronavirus pandemic for students in school during the pandemic.
• How to apply residual balancing adjustments in the light of the solution to previous bullet.
• Whether in the light of the Review the weights used in the cohort-split model needed to be updated.
• How to handle missing data in this new model.

Previous Years:

The existing model works to retrospectively apportion attainment outcomes across all years spent in education instead of taking it all from one single year’s attainment score. The ONS made an additional adjustment to the academic years affected by the coronavirus pandemic, namely academic years ending 2021 and 2022. Prior to the coronavirus pandemic, the attainment index score for a given year would be back-cast and apportioned between year groups in the back-series e.g. X% of the score goes to Year 7, Y% of the score goes to Year 10 etc. However, during the pandemic years, the attainment index score has not been back-cast in the typical manner, but rather year groups prior to academic year ending 2021 are now calculating using the average performance of respective year groups over the previous five years. The Review considers this provides the most reasonable estimate of likely attainment at this time in the absence of final attainment data unaffected by the coronavirus pandemic for these year groups.

However, this imposes a constraint on the system: by effectively fixing the value for one or more years, this means the total contributions from within a cohort affected by the pandemic will not automatically equal the attainment value achieved when that cohort eventually sit their GCSEs. It should be noted that the NRT factor described above similarly ‘fixes’ the values in academic year ending 2021, implying that residual adjustments will apply to all year groups who were in school during the pandemic, but did not sit their GCSEs in that year.

Residual Adjustments:

Therefore, residual adjustments need to be made to year groups affected during the coronavirus pandemic to balance the sum of annual contributions to the final achieved attainment value for that year group. Residual adjustments are only applied from academic year ending 2021 onwards. For the first cohort affected by the coronavirus pandemic (Year 11 in academic year ending 2021), the Year 11 score needs to be adjusted so that the contributions of Year 7 to Year 11 equal the attainment index value. To prevent arbitrary application of residual adjustments in different cohorts in the same year, the Year 11 residual adjustment in that year is applied to the other year groups. The same process is then followed for the subsequent years taking previous residual adjustments as given. These adjustments will automatically drop out of the system when the last cohort who were in school during the coronavirus pandemic sit their GCSEs.

Cohort Weights:

Having developed this method, in discussion with experts a follow-up question emerged, which the Review has also addressed, namely the value applied to the weights placed on each year in the cohort-split model.

Within the model, primary and secondary schools were adjusted separately, each with their own corresponding weights and these adjustments were performed for all four nations of the United Kingdom. Whilst the seven years of primary education (Reception plus Years 1 to 6) received equal weights of 14.3% each, secondary schools cohort scores diminished with distance from sitting GCSEs, (Table 8), so most weight was given to those years closest to the achieved scores. Equal weights (25%) were also applied to Further Education (FE) when this was added into the model for age groups 16-19. FE attainment will be factored into PSP estimates for the first time in Spring 2025.

The Review considered the secondary school weighting for revision because the model appeared inconsistent with primary and FE, and because attainment was apportioned back to Reception, when these years had already been captured in the primary attainment. For these reasons the new weights laid out below are proposed for implementation into the cohort-split model. Following discussion with DfE, equal weights (20% for each year) was given to the years 7 to 11 because this approach was felt to provide a reasonable split across the secondary education in absence of robust literature review or additional data.

Table 8: Weights for each school year’s contribution to attainment

The review proposes this setup based on the following criteria:

• It allows for consistency in the weighting design between all school phases, in that weights are allocated to year and age groups specific to that sector.
• The Review acknowledges there is a paradigm shift between school phases, which supports the notion that achievements in the previous phase serve as a gateway to the next phase and new knowledge begins accumulating from this point.
• The main outcome of Primary schooling are exams sat at Year 6 (for example, SATs in England and Wales), therefore primary year groups weighted contributions need to build exclusively towards this.

Missing Data:

It must be acknowledged that the coronavirus pandemic caused unprecedented, widespread disruption to education, which has severely limited capacity to accurately measure output. This problem affects different settings in the different devolved governments.

For primary schools:

• England: No data have been published for academic year ending 2020 and 2021.
• Wales: No data have been published since academic year ending 2019.
• Northern Ireland: No data have been published since academic year ending 2019 – The Northern Irish Department of Education have confirmed they are reviewing the design of key stage assessments following the coronavirus pandemic, and that data for key stages 1 to 3 will likely not be available while the Review takes place.
• Scotland: No data have been published for academic year ending 2020.

For further education:

• England Level 2: The index between academic year ending 2020 and 2025 will need to be kept constant due to inconsistent grading practices during this time, and attainment can be factored back into the model from academic year ending 2026.
• England Level 3: The index between academic year ending 2020 and 2023 will need to be kept constant due to inconsistent grading practices during this time, and that attainment can be factored back into the model from academic year ending 2024.

There is no ‘perfect’ approach to the data gaps issue, and there are caveats regardless of the approach.

Whilst the NRT was sourced from secondary schools and hence could be directly applied, the same issues applied to primary and FE phases, but thorough discussions with DfE and other experts could not reveal NRT-equivalent data for these phases. Despite the difficulties related to applying the NRT outside of secondary school, as the NRT is a secondary school measure and analysis conducted by the ONS indicated that the trends in the NRT do not show synergy with trends observed in primary schools, in the absence of alternative data, using the NRT to indicate attainment trends in other school phases remains the only option.

The Review therefore proposes to include the NRT to fill in data gaps for remaining primary schools across all four nations of the UK. There is also the question of whether FE also needs to be treated in a similar manner. Unlike Primary and Secondary education, FE does not have as much of a focus on academic-orientated subjects and does include more technical-orientated learning.

Attainment for Further Education

The Review has also expanded the quality adjustment to FE, to deliver coverage of all years of compulsory schooling (ages 3 to 19 years), which has been mandatory since 2013. Through consulting with the DfE, the Review was able to identify the National Qualifications Framework (NQF), which is a structured and consistent foundation for reporting the different qualifications sat by 16 to 19-year-olds in FE. The NQF also has good sustainability for future research, as the further introduction of new qualifications will be compatible with the existing framework.

The Review identified Level 2 and Level 3 attainment at ages 16 to 25 as a data source, this is based on the percentage of students who achieve the minimum grades at each level, with coverage from academic year ending 2004. Level 2 (L2) qualifications refer to GCSE level qualifications or equivalent technical qualifications. In addition, students may re-sit Mathematics and English GCSEs throughout FE, allowing this relevant output of FE to be captured in the estimates. Level 3 (L3) qualifications refer to A level qualifications and vocational qualifications, such as NVQs.

The growth rate in attainment from this source serves as the basis of the ONS’ FE attainment measure, using the percentage of students who achieve the minimum required grades for L2 and L3 by age 19 only. This is because:

• Students will have sat their final exams and moved through the FE system by age 19.
• L3 attainment at age 16 is low (it is uncommon for students to complete L3 qualifications by age 16), and L3 attainment at age 17 drops significantly in academic year ending 2016 with the exclusion of the contribution of AS levels to final attainment at L3.
• L2 attainment by age 19 accounts for students who resit their GCSE Mathematics and English examinations throughout FE.

The review proposes to treat FE as a standalone component in the cohort-split model; in other words, FE attainment will be retrospectively apportioned to age groups exclusive to FE (age 16 onwards) and not to previous school phases (Primary and Secondary schools), based on consultation with DfE, Ofqual, and the Institute for Fiscal Studies. The reason for this is because:

• Students often move to a different sixth form or college to their secondary education, and FE has different funding mechanisms and organisational frameworks to secondary education. Therefore, FE is more distinctly isolated from previous school phases, and this makes it difficult to link FE performance to previous years of schooling in a consistent manner.
• Students may not sit academic qualifications at FE, instead they may pursue vocational certificates which do not draw on prior knowledge to the same degree as academic qualifications.
• GCSEs serve as a gateway to entering FE, therefore knowledge at the FE level starts at age 16 and builds from this point.
• Attainment at age 19 is reported by age group, as opposed to academic year groups (e.g. Years 12 and 13) which are reported by Primary and Secondary schools. Consequently, there would be difficulties in integrating age groups in FE to academic year groups in previous school phases.

Upon further discussions with experts, it was also agreed that each age group being accounted for in the cohort model (ages 16, 17, 18, 19) will bear equal weight (25% each). Currently, there is no compelling or robust enough evidence to clearly indicate that one age group has more bearing on achieved attainment than others. Therefore, until such evidence emerges, equal weighting will be applied to these age groups.

As two separate attainment indices (L2 achievements by age 19, and L3 achievements by age 19) will be informing the FE attainment measure, there is a question on how these will be weighted and aggregated into a single index measure for FE attainment following cohort model adjustments for each. The Review explored avenues such as weighting by the relative cost of delivering L2 and L3 qualifications, enrolment figures for each qualification type, or the employability potential of each qualification. Following workshop discussions on the topic with DfE, it was agreed that the enrolment figures for students completing L2 and L3 qualifications by age 19 each year were deemed the most appropriate and empirical metric to determine the weights. It is typical across PSP to weight components based on their cost of delivery, however the total costs for providing education according to qualification type will be directly linked to enrolment figures. The figures vary on an annual basis, but the proportions of L2 and L3 qualifications achieved each year are roughly 60% and 40%, respectively.

Student Well-being

In addition to including attainment in the quality adjustment, the ONS has historically included bullying rates as an additional measure. The Review has identified that this does not fully represent the wider impact of schools on their students beyond academic performance. For this reason, it has been replaced by a more inclusive measure of well-being in schools, which is seen to better reflect different aspects of the development of young people into being capable members of society. Student well-being has also been identified as a more appropriate quality adjusted measure than bullying because of its association with academic achievement and school engagement (Gutman & Vorhaus, 2012). An increasing number of policies focusing on well-being were also implemented in the education service after the coronavirus pandemic, reflecting a growing focus in this area.

The Review identified in the Understanding Society’s harmonised UK Household Longitudinal Survey (UKHLS) and its predecessor, the British Household Panel Survey (BHPS), as its preferred data sources. The BHPS ran from 1991 to 2009, while the UKHLS commenced in 2009 and continued sampling from households that were included in the BHPS. These surveys represent the entire United Kingdom. The student wellbeing index is created based on weighted responses on the following questions: how do you feel about your ‘school?’ and how do you feel about your ‘schoolwork’?. The possible answers range from 1 (completely happy) to 7 (completely unhappy), with the midpoint 4 representing neither happy nor unhappy. Those who provide positive answers (from 1 to 3) to both the ‘school’ and schoolwork’ questions are flagged as ‘happy’ in each year from 2003 onwards to derive a wellbeing index.

Because well-being is only one component of the total quality adjustment for Education, the ONS need to derive an appropriate weight for well-being compared with the other components. It is common across service areas to weight components based on their delivery cost. However, in the absence of an alternative metric, the Review proposes using this approach.

The challenge is that it is not straightforward to identify expenditure that is solely attributable to the delivery of activities directed at well-being in education. For example, expenditure allocated to breakfast clubs has a link to well-being, but delivery of these activities also correlates with improved performance and attainment. However, the National Funding Formula (NFF) was identified as the most relevant data source to provide cost-associated weights for well-being.

The NFF is the process by which the DfE decides how much core funding to allocate to mainstream state-funded schools in England. Specifically, the ONS is using planned expenditure share (%) allocated towards funding pupil deprivation to proxy for expenditure targeted at pupil well-being. The association between ameliorating pupil deprivation and improving satisfaction and emotional experiences for students implied that expenditure allocated to this space was an appropriate means to weight the well-being measure, in the absence of explicit expenditure targeted at “well-being programmes”. The NFF was introduced in academic year ending 2019, so there are only observable weights from this period.

For the years 2003 to 2017 (the well-being measure commences in 2003), the ONS has held the weights constant with the earliest known annual weight. Due to weights being provided on an academic year basis, the ONS is required to spline the expenditure shares to generate figures in calendar year terms (consistent with periodicity of the annual statistics). The calculated weights assign roughly 9% to well-being; it must be acknowledged that well-being serves as a precursor to the primary outcome of education, which is attainment, and there are factors beyond the control of schools which can impact a student’s response to the survey. Therefore, the well-being measure cannot be assigned too much weight, and it was agreed between the ONS and DfE that the weights provided by the NFF for well-being are appropriate and fit for purpose.