Methods and Quality
There is no single approach that defines qualitative research. Different fields, disciplines, and even individual researchers have different approaches to qualitative methods. Researchers should consider the ethical challenges associated with any methods and the subsequent analysis that they choose to carry out, to ensure that the method(s) chosen best help to answer the research questions being posed. The research method that you choose will affect the type of data that you collect. Below, we outline some of the common qualitative methods used within the ONS. Whilst this is not an exhaustive list of methods, we have provided a brief overview of each method, and when they are often used.
Interviews are one of the most used qualitative data collection methods. They can be conducted online, face-to-face, or via the telephone. Qualitative interviews typically involve discussion between one researcher and one participant (though occasionally, more than one participant or researcher may be present). The discussion is guided by the researcher, based upon the aims of the research and the topic of interest. There are several different ways to structure interviews, dependent on the aims of the research and the topic of conversation. Interviews can be semi-structured, structured or unstructured.
Semi-structured interviews: Semi-structured interviews utilise a set of key questions, chosen based upon the research aims. This is known as a topic guide. It provides some guidance on what to talk about and allows the interviewer room to ask probing questions where necessary. It also gives the e participant an opportunity to talk about topics which the researchers may not have thought about.
Structured interviews: Much like semi-structured interviews, structured interviews typically utilise a topic guide. However, questions are completely predetermined with little flexibility or variation for interviewers to ask follow-up or probing questions. Questions are asked in a set order, and there is no variation in the questions asked to each participant. Asking set questions, in a specific order helps researchers to easily compare responses from participants. However, it also limits the data collected from participants, as neither party has the flexibility to move away from the set questions being asked.
Unstructured interviews: Unstructured interviews (also known as non-directive interviewing) are typically very informal and flexible. Unlike structured and semi-structured interviews, they do not utilise a topic guide, and questions are not decided in advance. Unstructured interviews are less common than those described above but are a useful tool in exploratory research. The flexibility enables researchers and participants to delve deeper into different topic areas and issues relating to the study.
When to use: Interviews are a suitable method to use when you want to better understand the views, beliefs, or experiences of participants on the topic of interest. Interviews can produce in-depth discussion, allowing for a topic to be explored in detail. The type of interview structure chosen will greatly affect the data collected, so it is important to consider this in relation to the aims of the research study and the needs of potential participants. Whilst an unstructured interview may produce more detailed or wide-spread data for example, a semi-structured interview may be easier to conduct, and provide more specific data relevant to the study. Whilst interviews are a very popular qualitative method, they do have their limitations. For example, conducting and analysing interviews can be time-consuming. This should be taken into consideration at the research design phase.
Cognitive interviewing is a method used to provide insight into an individual’s perceptions and understanding of a product or service. Traditionally, cognitive interviewing has been used to test participants’ comprehension of survey questions. However, it can be applied more widely. Cognitive interviewing often uses a ‘think aloud’ approach. This invites participants to talk through their thoughts and feelings as they perform a task or use a product. This helps the researcher to understand how the respondent interprets and comprehends the task, and the thought-processes behind their decisions.
When to use: Cognitive interviewing can be used to understand a respondent’s:
- Comprehension of text (e.g survey questions)
- Recall processes
- Judgement of the information given to them
- Ability and willingness to do the task being presented
Cognitive interviewing can help improve accessibility and inclusivity, ensuring that materials are developed with the user or participant in mind, thus improving the quality of the data being collected. Cognitive interviewing can be used in the design and development phase of research, or when you already have a product that you would like to improve.
Usability testing is a form of interview where participants are asked to complete specific tasks relating to the service being studied, to help understand how easy or difficult a product is to use. For example, it may be used to test prototypes or to understand problems in an existing service. This may involve participants completing a task whilst being observed by a researcher, to establish what works well and what features may be causing issues. Like cognitive interviews, usability testing may also encourage participants to use a ‘think aloud’ approach (see cognitive interviewing section for a definition).
When to use: Usability testing can be used to find out whether users understand how to use a service, website, or product, and to identify how it can be improved to ensure that it works as intended. In contrast to cognitive interviewing which explores participants’ cognitive understanding, usability testing focuses on the functionality of a service.
‘Cogability’ testing is a hybrid technique, combing elements of both cognitive interviewing and usability testing. It aims to test both the participants cognition (their understanding), and the usability of a product or service. It is important to explore these two aspects together as one can impact the other. For example, visual design and layout can influence comprehension. If they are not explored together in the same session, then you may risk changing the wrong feature in the design. Cogability testing also encourages participants to use a ‘think aloud’ approach (see cognitive interviewing section for a definition).
When to use: Cogability testing is often used in survey design, as it can help capture a participant’s understanding of the questions being asked, as well as their ability to navigate the survey questions and the survey’s format. Cogability testing can be carried out in person or online.
A focus group is a research method which brings together a small group of participants to discuss the topic being studied, and their opinions and experiences. Often participants will share a similar characteristic or experience related to the topic, and this will be the basis for the discussion. This can help researchers to determine where there is agreement and disagreement between different participant views and perceptions.
When to use: Focus groups are useful to understand a participant group’s views, opinions, and attitudes of a topic. There are many advantages to using focus groups in comparison to other methods such as interviews. Focus groups help to promote discussion, idea sharing, and debate. They can also be used to help determine the usefulness of a product or service. Focus groups can be used at many different stages of a research project. For example, they can be used:
- to help researchers explore points of interest at the research design stage
- as the primary method of data collection
- to test recommendations made at the end of a research project (or once a service or product has been developed)
- As an opportunity to use interactive, task-based exercises to gather group insights
Pop-up research involves conducting short, informal interviews or usability tests in places that the people who you want to talk to typically use. This could include libraries, office spaces, or shopping centres. Pop-up testing requires the researcher to recruit participants on the day itself, and participation is immediate. For this reason, activities used in pop-up research should be short and easy to explain. Because pop-up research limits the length of interaction with participants, it should always be used in combination with other research methods.
When to use: Pop-up research is usually done in the early phases of research to help guide the project on which option to take or path to pursue. It is not used to gather critical decision-making insights as it not rich enough. Pop-up research is useful for when you may need to:
- collect insights quickly
- talk to specific, hard to reach user groups
- conduct research in different places to understand regional differences
Participatory research is a group of methods which involve the collaboration of researchers and participants to better understand social issues. It aims to hand the power from the researcher to the participant by giving participants a bigger role in the research process. Participatory research utilises a range of activities that enable participants to explore and express ideas in a way that makes sense for them. For example, this may include the use of video-diaries, visual and creative methods, such as vision-boarding or poster-building exercises, and idea-mapping.
When to use: Participatory methods can be used in any part of the project cycle, including planning, data collection or analysis. This allows for participants to play an active role in the research and decision-making. Participatory methods are often used when working with children and young people, people with learning disabilities and groups or communities. These methods can be useful to create ‘distancing’. This means that the discussion of an issue centres not on the personal experiences of a participant, but the social, political, or environmental context of an issue more generally.
Ethnographic research describes the practice of observing people in their own environment. It aims to understand behaviours by viewing a person’s actions, attitudes, and emotions in their natural surroundings. Ethnographic research can be done in a number of ways, including taking field notes, document analysis, and filming. This makes it a very varied method of data collection and analysis.
A researcher may record participant behaviours by watching how they behave or interact in an environment, or by listening to conversations. Observations can be overt or covert, participatory or non-participatory and direct or indirect. In overt observations, participants are aware they are being observed, whilst in covert observations, participants are not aware they are being observed. In participatory observation, the researcher takes part in the activity, whilst in non-participatory observation the researcher is not involved in the activity. Direct observation involves the researcher observing an activity whilst it is happening, and indirect observation involves the researcher observing the results of an activity after it has taken place.
When to use: Ethnography can be used to understand how a particular group or community act within a specific context. Ethnographic research focuses on exploring experiences, rather than testing hypotheses. It can help researchers to understand not only what people may say, but also what they do, and how this compares.
Bias in Qualitative Research
Bias can be introduced into qualitative research at any stage of the research process. The possibility of bias should be discussed by researchers when planning their project. This will help to minimise bias before it becomes problematic.
One of the most common ways that bias is introduced into qualitative research is by the researcher. For example, researchers may have preconceived ideas or notions about the topic being studied. This may influence the questions that they ask participants, the way in which questions are asked, and how they interpret and analyse the data being collected.
Researchers should avoid asking leading questions when asking participants questions, as they may affect the results and credibility of the research. Leading questions may cause a bias in responses, often aligning with the views or goals of the researcher, as participants are encouraged or led to answer in a specific way. This may also dissuade participants from talking about topics which are unexpected and could also impact upon the relationship between the researcher and participant. Respondents may be likely to repeat words used by the researcher or talk more in-depth about topics that have a reaction from the researcher, as they may assume the topics is important for the research.
The importance of question structure – bias in leading questions
Leading questions may encourage a biased response, based on the way the question is framed. Researchers may not be aware that they are asking leading questions as they may be a result of unconscious bias. It is therefore very important that researchers are appropriately trained in their methodology. The possibility of asking leading questions can be mitigated by creating topic guides, and leading questions are less likely to occur in structured interviews where there is less variation and flexibility in the questions asked. Topic guides should be checked for leading questions before being used. Researchers may find it helpful to ask colleagues to look over the topic guides, as they may spot different biases in the structure or wording of the questions. In methods where researchers may move away from the topic guide to probe the participant more, it is also important that researchers remain as neutral as possible both when they ask questions, and when they respond to the information given by participants. This can be more difficult when questions are unplanned. Researchers should note down these interactions and be reflexive about any possible bias in the structure or content of the questions being asked.
Be aware of your influence as a researcher
It is also important to reflect on whether the presence of a researcher will influence how individuals may act. For example, participants may behave differently if they know they are being watched, or if they think that the researcher wants them to act in a certain way. Whilst this can be difficult to control, it is important that researchers communicate the potential implications of this bias when they disseminate their research. Reflexivity is key. It is also important that researchers build rapport with their participants, so that participants feel comfortable, and trust the researcher they are interacting with.
Positionality considers a researcher’s relative social, cultural, and political location in relation to another person in a particular context. Positionality is closely related to a person’s social identities, standpoints, and cultural practices. As a researcher, you should consider how you see yourself and how you are viewed by others, and then how this may affect your data collection. For example, the clothing and sociodemographic of the researcher can impact data collection. Researchers should think about the degree of connection to the participant group you are working with, and the positive and negative consequences this may have. You should be mindful of assumptions, for example how difficult or easy it is in creating rapport and gaining insight from your participants. Being aware of what is happening and why will help to increase validity of your results.
A note on group-based research
Group dynamics will have a significant impact on the quality of the data that is collected. For example, whilst focus groups give participants an opportunity to engage in the discussion of shared experiences and debate, it can be difficult to ensure that the discussion remains relevant to the topic being studied.
It is the role of the researcher to ensure that discussion is kept on track. It is also important that participants can voice their opinions, and that discussion is not unnecessarily stifled. It can be difficult to ensure that no participant dominates the conversation, and that quieter voices still feel valued. It is therefore important that the researcher acts both as a moderator and a facilitator when conducing focus groups. The methods for doing so may vary depending on the mode of the session. For example, body language can be used in-person focus groups, but when sessions are being conducted remotely the researcher may have to find other ways to do so.
Researchers may wish to use topic guides to ensure that the discussions stay relevant, whilst allowing the flexibility for participants to talk around the topic. It is also important for the researcher to create a safe space where participants feel comfortable to share their opinions, managing discussions to avoid confrontation and distress between participants. Researchers should lay out a set of rules for participants to abide by during the focus group. This may include asking participants to ensure that they do not speak over each other and that they are respectful of other people’s opinions.
You may also need to consider the possibility of harm from over-disclosure, distress, or embarrassment in a group setting. You will also need to be aware of vicarious re-traumatisation, especially when discussing sensitive topics in a group setting. Although one participant may be comfortable discussing and sharing a potentially sensitive experience, this may be detrimental to another participants wellbeing.
Reflexivity
Qualitative research should be reflexive. Reflexivity involves examining how your own judgements, thoughts and feelings, and biases may affect a situation. Unconscious bias is making judgements or decisions based on prior experiences, assumptions or interpretations, without being aware you are even doing this. As a researcher, you should take the time to consider how this may influence your research, including your data collection and analysis.
Researchers play a fundamental role in qualitative research and can actively influence research outcomes. Any underlying beliefs or behaviours, both positive and negative, can impact data collection. Once researchers have collected and coded their data, themes and relationships can be identified. You can question your own assumptions to understand whether you hold any beliefs which may have led you to a particular conclusion. Reflexivity can help identify whether any of your predisposed beliefs, opinions or biases may have impacted the research. It can be helpful to log decisions during your research to enable you to be reflexive throughout your project.
Incorporating reflexivity in your research and findings allows you to be open and honest. It can provide clarity to your results and help stakeholders understand how you came to your conclusion. This doesn’t need to be every detail, rather the researcher’s relationship to the research and why certain outcomes may have come about.
Back to topMitigations and Advice
- Take time to reflect on possible biases that may find their way into the data you are collecting and using. Any potential for bias, no matter how big or small, and the impacts that may arise from this should be documented throughout the project. Reflexivity on the part of the researcher is key.
- It may be useful to discuss your project with colleagues as part of an independent review. This may help to determine whether any biases have been introduced into the dataset or results.
- You should consider how the topic that you are researching may impact on the participants that you recruit, and how harm can be minimised. This should be discussed when designing your research project, as it may have an effect on the methods that you choose to use.
- It may be beneficial to consider how Respondent Centred Design can be applied to you research project to ensure that the needs of respondents are identified, and methods are designed in a way that meets those needs.
- As a researcher, you should be aware of and acknowledge your bias. This can be valid at all points of the research project, from defining your research question, selecting your methodology and mode, data collection, analysis, and dissemination. You should be aware of your position as a researcher, and how your background may affect data collection and analysis. Researchers might find it useful to produce a clear statement of why their conclusions are believed to be valid, what is meant by ‘being valid’ for their particular use case, and any limitations on the validity.
- It may be beneficial to have multiple researchers involved in data collection, coding and analysis, to ensure bias is avoided and there is agreement in any interpretations.
- Researchers may wish to ask participants to review transcripts and reports, to ensure their experience and opinion has been correctly captured or interpreted.