“Objective” and “Unbiased” Qualitative Research

MCL Guidance

By Dan Dohan

Scientists like numbers. Words and narrative can help illustrate or describe research results, but scientists typically expect trustworthy results to take a numeric form. Well-trained researchers recognize that quantification does not assure unbiased objectivity. Nevertheless, in scientific communication, numbers enjoy the presumption of innocence while narrative triggers suspicions of subjectivity and bias. How can qualitative data from a small number of people be scientific? How can you interview someone without bias? What about the Hawthorne effect? Without statistics, how can you be sure your findings are real? How can quotes constitute trustworthy results?

Sensitive to these questions, some qualitative researchers respond by striving to adopt quantitative techniques in their qualitative studies. They draw large interview samples and highlight their representativeness. They develop guides to structure interviews or record observations. They train data coders to use a standard approach as they apply a code to each segment of data. They present data in tables; each quote represents a theme; each quote represents a particular type of respondent. Reviewers grade how well studies’ methodology aligns with a standard set of protocols that have been adapted from quantitative science.

Other researchers point out, however, that the emperor of objectivity has no clothes. They note, accurately, that science is done by humans, so subjectivity and bias are inescapable. Quantifying techniques, e.g., sampling, surveys, and statistics, may address these limitations to some extent but are no panacea. They do not make science objective and bias-free.

In addition, there are trade-offs when a qualitative study embraces a quantitative logic. On a positive note, when qualitative work embraces standardization it can bolster its legitimacy among quantitatively-oriented disciplines; the trade-off may be necessary to reach certain scholarly communities. But drawing random samples, standardizing data collection, tabulating rather than interpreting results, and summarizing findings in tables and figures — these run counter to the qualitative scholarship tenet of conveying the world through the eyes and experiences of participants. Conversely, scholars who ground their work in championing subjectivity and rejecting the possibility of objectivity encounter their own trade-offs.  They find themselves in a bind, having lost the ability to dialog with a broad swath of the scientific community.

There is a third way to engage the challenges of objectivity and bias in qualitative research. It neither embraces the quantitative terms of debate nor rejects the value of striving for objectivity. It starts by recognizing that all researchers must address four fundamental questions:

(1) How do I select research subjects? (data collection);

(2) How do I work with subjects to get data? (data collection);

(3) How do I generate defensible findings? (data analysis);

(4) How do I convince readers? (data analysis).

As the list highlights, these questions fall into two familiar buckets. Selecting and engaging subjects occurs during data collection. Identifying and communicating findings are products of data analysis.

Quantitative scientists answer these four questions with R words: drawing Representative samples, minimizing Reactivity during data gathering, generating Reliable statistical results, and emphasizing their potential Replicability. These terms dominate scientific discourse. Many assume that representative samples responding to standardized instruments is THE scientific method for data collection. That statistical analyses and the tables and confidence intervals they generate are THE scientific way to present data.

This is not true. Returning to the four tasks of scientific research helps us recognize the invaluable contributions of qualitative research to scientific discourse. It helps us understand how a scholarly community can strive for scientific rigor and reproducibility without using the four R’s.

Selecting participants

Qualitative researchers typically invite people to join their study in a purposeful fashion. Purposeful sampling requires thoughtful design. Designs lead investigators to the proper places and people to recruit. Investigators refer to the design when they describe their value in the study to potential participants. Thoughtful designs also respond and evolve over the course of the work. New data generates new insights and new questions which sends investigators to new places or people or inspires new questions to ask of current participants.

Compared to a purposeful sample, a sample that is merely representative is, at best, inefficient. Representative samples are mechanical. They aim to quantify the presence (or absence) of something measurable within a pre-defined group. In contrast, a purposeful sample enables the researcher to classify, describe, and understand variety. It seeks to document the range of behaviors, opinions, or situations that are relevant to the research question.

Gathering data

Qualitative researchers participate with subjects to gather data. They conduct semi-structured interviews, convene focus groups, and hang out in the field. They select a technique to fit the situation and data needs of the project. Talented fieldworkers shape questions and interactions to respondents’ sensibilities. A flexible participatory approach helps researchers appreciate and understand subjects’ experiences of the world. In contrast, non-reactive approaches record participants’ responses to pre-set research procedures.

The participatory approach means that appearance, identity, and lived experience inevitably shape how fieldworkers and subjects engage each other. Humility and self-reflection are an essential condition of fieldworkers’ craft. But self-reflection can only go so far. In an ideal world, projects enter the field with a diverse team of fieldworkers who respectfully and productively engage participants. Qualitative studies use participation – specifically, the diverse ways researchers engage subjects – to gain insights into their worlds.

Generating defensible findings

Quantitative scholars defend their findings with statistical reliability. In contrast, qualitative scholars produce defensible findings by iteratively and patiently processing their data. They review and critically reflect while collecting it. They identify when and how data seem consistent (or not), and they use those insights to guide ongoing data collection. As the process continues, some findings firm up; they are readily defensible. Counterfactual findings crystalize. To understand and potentially turn those counterfactual findings into defensible ones may require new sites, informants, or techniques. 

These iterative processes align with the well-known grounded theory (GT) approach. But qualitative findings are defensible even if they do not use GT-style inductive reasoning, coding, and theory building. The well-choreographed GT approach – and especially the GT notion of saturation – can sometimes be mistaken for a type of reliability test. But the magic components of GT and other qualitative analyses are the processes of inquiry, reflection, and refinement, not a predefined technical procedure or outcome.

Convincing readers

Quantitative scholars establish credibility by presenting findings in ways that suggest they are readily replicable. Replicability has long been touted as a foundational principle of science. But even in the quantitative social sciences, it can prove easier said than done; few scholars attempt to replicate others’ results and it can be elusive when attempted. Qualitative data collection is not designed to support replication; the study of a purposeful sample examined via participatory data collection can be done only once.

Qualitative authors do not rely on replicability to convince their readers. Rather, they present data with particularity. Data presented with context and forthright detail establishes legitimacy. In a way, qualitative scholars convince readers using the oldest trick in humanity’s book: they tell a believable story.

As noted above, scientists view stories with suspicion. Tidy emplotment may satisfy lay readers, but it can provoke skepticism among scholars. Qualitative results are more convincing when quotes and observations in the results section encourage active and critical thinking. Details should illustrate how context matters; this helps avoid the trap of appearing to over-generalize qualitative findings. Sharing counterfactuals – and their resolution – can help readers recognize the analytic process that generated findings.

The table below summarizes the ways quantitative and qualitative traditions may approach the four basic questions of scientific research. This table illustrates that both traditions have developed a strategy to guide the logistical tasks of collecting and managing data. Both traditions have found ways to analyze and present data so their work conveys its own expression of rigor. Both traditions have done so in ways that are consistent and coherent. Embracing R’s and P’s thus provides researchers with a coherent approach for navigating the four challenges of research. On the other hand, combining R's and P’s in single project may be self-defeating or nonsensical. It is not impossible, but must be done with careful consideration.

Research Tasks

R’s (Quant)

P’s (Qual)

Data Collection

Select subjects to include

Representativeness

Purposefulness

Work with subjects to get data

non-Reactive measures

Participation

Data Analysis

Generate defensible results

statistical Reliability

interactive Process

Convince readers of findings

Replication

Particularity

Striving to conduct qualitative research in an objective and unbiased fashion is a worthwhile ambition even if, ultimately, an impossible goal. Yet, it is important to do so thoughtfully. Embracing quantitative science’s strategy at best distracts from — and at worse prevents researchers from realizing — the unique strengths of qualitative research. Understanding the ways high-quality research looks similar and how it differs across scholarly traditions will benefit us all.