23 June 2008
Methodology, Assessment and Service Delivery
It is a great pleasure to have been invited to make my first public comments as Head of Assessment, at this year’s GSS Methodology conference.
And for reasons I shall come to I think it is appropriate that we should seek, at an early stage in the development of the new Assessment function, to open up a dialogue between the Statistics Authority and all of you who are interested in matters of methodology.
I have always regarded methodology as the hard stuff of statistics and best left to people who know a lot more about it than me.
Consequently I am not quite sure of the boundaries around the field that we call “methodology”.
I know some of the things on the “inside”, not least from studying the programme for today’s conference – including time-series analysis, projections, index number construction, survey methods, disclosure control, classifications, small area estimation, the measurement of output and productivity, and issues of statistical geography.
But, I wonder if – in the rapidly evolving world of official statistics – the boundaries now perhaps need to be wider still. And this is a theme I want to explore with you a little further in a minute and as part of that ongoing dialogue I mentioned a moment ago.
On that theme, I would like to pay tribute to superb and responsive advice that I received when I was chief executive of the Statistics Commission from methodology colleagues in ONS, and I’ll embarrass him by mentioning Peter Goldblatt in particular.
But at least I start in the new arena of Assessment with an acute sense of how important it will be to turn to methodologists for advice – on rather a lot of issues. Let me be absolutely clear on this point. The Authority’s Assessment function will not be setting up in competition with current centres of expertise. We will rather be your most enthusiastic customers.
I’d like to say a little more about what Assessment means and the link with methods and methodology.
The Statistics and Registration Service Act defines Assessment quite narrowly. In the terms of the Act, Assessment is essentially a matter of first examining chunks of statistical activity, to see if they are compliant with a revised Code of Practice – a code which the Authority will itself approve – and then producing reports: we are calling these “Designation Reports”.
On Friday of last week, the Authority agreed the text of a consultation document including our proposals on a revised Code of Practice and we will publish this soon. We really do want to know what you think of it, so please feed in your comments as part of the consultation process.
The Code serves, in effect, as a ‘contract’ between the Authority, on the one hand and organisations that produce official statistics on the other – setting out what is expected of them in broad terms.
And the Code also serves as a contract between the Authority and Parliament – setting out what Parliament can expect the Authority to pursue on its behalf. So the Code is a critical piece of the new statutory infrastructure.
Producers of National Statistics are under a statutory duty to comply with the Code. The Assessment process, therefore, is more than just a source of advice but I am determined that it will be constructive, helpful and forward looking. Some colleagues in the GSS seem concerned that it will be used to criticise them. I will take this and every opportunity to stress that that is not the aim. The aim is to help push forward improvement in the service in the public interest.
Principle 3 of the revised Code – which is much shorter than the current National Statistics Code of Practice – says that
“Methods for the production, management and dissemination of official statistics should accord with scientific principles and internationally recognised best practice” and
“quality should be monitored and assured taking account of internationally agreed concepts of statistical quality”.
As I’ve said the staff who will carry out the Assessment of chunks of statistical activity will not themselves be expert methodologists. Nor would I want them to be.
We will therefore, through some means yet to be fully worked out, need a formal mechanism for inviting appropriate experts to assist us in approving the methodological aspects of statistical work in all the bodies that produce National Statistics, throughout the UK. Again, this must be part of an ongoing dialogue.
You may be thinking that this means additional work for you, and I think it does too, but it also gives methodologists more practical authority across government than ever before. And that must be a good thing.
However, this regular Assessment work, leading to Designation Reports, is only half the story.
The Statistics Authority will also be producing a long term series of one-off reports on ‘issues’ in official statistics; and we have imaginatively labelled these as “Issue Reports”.
The consultation document on the Code of Practice will itself be the first of these issue reports. After that, the Authority’s issue reports will cover both major matters of public concern, and also more process or organisational questions.
I can tell you today that the Authority will this year be starting work on reports on 1) the communication of inflation and consumer prices, 2) on the progress being made to improve migration statistics and 3) on statistics not currently within National Statistics which should be and 4) barriers to trust in relation to crime statistics.
We have also decided on some of a second wave of reports to include 1) the arrangements needed for long term planning for the statistical service as a whole to meet the longer term needs of society, and 2) the adequacy of environmental statistics to inform public debate. We do, however, retain the option to juggle these and other reports to take account of our capacity and the views of various stakeholders.
I think however I have told you enough for you to see that we are setting our sights on the big issues…and will need all the help we can get.
So, what role for methodologists on these, high profile, one-off reports?
My guess is that the role will be substantial. Both in terms of advising the review work directly, helping to formulate conclusions and recommendations, but also helping with the actual implementation of the recommendations.
A report might for example recommend that methods be improved without being all that specific as to exactly what methods should be used. Implementation of the Authority reports is likely to become a significant area of work.
Now, just in case you are getting the impression that I may be pushing a lot of work your way, I want to add some more.
I want to return to the thought that the boundaries of what we call ‘methodology’ may not be drawn wide enough. I look to you for guidance on this.
In some sense, the existence of methodology is what defines statistical professionalism and distinguishes it from the efforts of non-professionals. Consequently, we all know we must follow best methodological practice when designing surveys and constructing indices, and so on.
But I wonder if the time has come to push the boundaries more and ask if we are following best methodological practice when, firstly, we consult users of statistics about their needs; secondly, when we extract the messages from the statistics in a consistent way; and thirdly when we communicate, via paper and the web, the finished product to users. I do recognise that both analysis and dissemination are on the agenda today but what I have in mind is not just the more technical aspects but rather the entire activity of extracting and communicating statistical messages.
In my view of the statistical system in the UK, viewed in the round, we are good at the core tasks of collecting and producing statistics but the areas I have just mentioned are our weaker one – the planning of what gets produced in the first place; the derivation of the statistical messages, and the communication of those messages to the user.
The new legislation gives the Authority the job of promoting the production and publication of statistics that “serve the public good”. In practice, that means statistics that serve the needs of people who need statistics to inform their work and actions.
So I will leave you with a question. Has the time now come where we should start to see the entire processes of planning, production and communication as professional matters to which professional methods should routinely apply?
I look forward to continuing discussion of these issues with you.