Principle 3 (Methods and Quality): The risks and limits of new technologies are considered and there is sufficient human oversight so that methods employed are consistent with recognised standards of integrity and quality
10. Validity
Low risk:
Confidence that the methods used, and quality of data will lead to valid conclusions
Average risk:
There is limited confidence/it is unsure whether the methods used, and quality of data will lead to valid conclusions
High risk:
Potential that methods used, and quality of data may/will lead to invalid conclusions
In many cases, you might use a dataset without knowing the quality of the data, the methods used to collect, process and visualise the data, and any assumptions made during those processes. All these factors may compromise the validity of the research. You should therefore strive to meet recognised standards of data quality and clearly state any hypotheses and assumptions.
Back to top11. Standards
Low risk:
The research organisation has established and tested procedures, and complies with recognised standards
Average risk:
There is limited confidence/it is not clear whether the organisation has established and tested procedures, and complies with recognised standards
High risk:
The research organisation does not have established clear procedures or may not comply with recognised standards
Compliance with recognised standards does not only ensure the validity of the research, but also the reproducibility of results. It improves the resilience of the organisation to public scrutiny and is a vital part of building public trust and confidence. Apart for auditable research procedures, researchers should have policies in place to assure the security of the research environment, for example, to manage data breaches.
Back to top12. Training
Low risk:
Researchers are appropriately trained to recognised standards
Average risk:
Researchers are trained but have limited experience in particular research area
High risk:
Researchers are trained but there is limited assurance in training that relates to this research
It is essential that researchers have an updated training portfolio over a broad spectrum of research skills and experience. Documenting these skillsets within the research team enables for more flexible working and ensures continuity and knowledge transfer. Organisationally, this provides assurance that, apart from the technical systems, staff have the required expertise to undertake the research specified.
Back to top13. Human Oversight
Low risk:
Human oversight of all elements of research process, and regular audits of automated outcomes
Average risk:
Significant use of automated processes and/or off the shelf solutions with some level of human oversight
High risk:
Research based on automated or opaque processes with minimal human oversight
The extended use of ‘off-the-shelf’ software solutions, and the use of code sharing platforms, requires you to be vigilant of assumptions and constraints which may not always be documented. Human oversight is a critical safeguard of any research governance process, requiring an emphasis on the quality of methods used, especially as automated processes become more opaque.
N/A: Omit this item in case of fully transparent automated or manual processes with well documented assumptions.
Back to top14. New technologies
Low risk:
Research utilises well established methods and technologies
Average risk:
Methods and tools may be tried, but are still novel or automated
High risk:
Research utilises untested or automated methods and technologies
Established methods and technologies have been tested extensively over long periods, are well documented, and have been subjected to scientific scrutiny. This offers assurance to the public that personal data are handled safely and provides confidence in the quality of research/statistical outputs. New technologies may entail a wider variety of unforeseen risks, from security to methodology, which may not have been discovered. Of course, the research community draws on innovation and should not miss the opportunity to transition to new technologies. Researchers should remain vigilant of the data sources and methods used in their projects and make sure that adequate security arrangements are in place.
If you are using location data or machine learning, see our guidance on ethical considerations in the use of geospatial data, and machine learning.
N/A: Omit this item for small-scale exploratory projects and feasibility studies which are not used to produce any research/statistical outputs.
Back to top15. Potential to realise benefits
Low risk:
Methods and quality of data will most likely result in realising the research benefits and fully mitigate any risks
Average risk:
There is limited confidence/it is unsure whether the methods and quality of data will result in realising research benefits or mitigate risks
High risk:
Methods and quality of data have little/no potential to result in realising research benefits or mitigate risks
It may not be enough to state the public benefit of your research project; you also need to make sure that the methods used, and the outcomes derived, can be used to realise the public benefit. Complex statistical outputs, increased number of assumptions, or the level of granularity and geography might not properly inform the public or decision-makers.
Back to top