By Mario Luis Small and Jessica McCrory Calarco, authors of Qualitative Literacy: A Guide to Evaluating Ethnographic and Interview Research
Suppose you are given two books, each based entirely on one year of ethnographic observations and interviews, and are told that one of them is a sound piece of empirical social science and the other, though interesting and beautifully written, is empirically unsound. What criteria would you use to tell the difference?
This question is relevant not only to the scholars who conduct qualitative research but also to a range of other readers, including those who formally assess qualitative research as evaluators and reviewers for universities, foundations, and academic presses. Additionally, scholars, policymakers, journalists, practitioners, and lay readers who care about the socially important topics that qualitative researchers have written on, need the skills to assess the validity of the research.
Essentially, this question is relevant to anyone who needs to demonstrate qualitative literacy—the ability to competently read, interpret, assess, and evaluate qualitative evidence. But as we argue in our book, many readers lack the qualitative literacy necessary to answer this question, creating a gap in public knowledge that is particularly stark given efforts to improve quantitative literacy in recent years. Moreover, those who do have the qualitative literacy to answer this question have yet to offer a set of criteria for identifying good qualitative social science.
Stepping into that gap, some scholars and practitioners trained in other methods of social inquiry—quantitative social science, legal studies, or journalism, for example—have suggested that criteria developed for their methods (e.g., generalizability, verifiability, transparency) should be applied to qualitative social science as well. But as we argue in Qualitative Literacy, empirically sound qualitative research is best identified using criteria that reflect what qualitative methods are optimal for.
Focusing on ethnographic and interview methods, we argue that these methods are distinct from other methods in that the researcher does not merely collect but also produce data, such that the data collector is explicitly in the data themselves. As a result, the nature and quality of the data collected in an ethnographic or interview study will, by design, depend on the in-the-moment decisions of the fieldworker as they respond and adapt to actions of the people they are interviewing or observing. Because of this, the challenges to producing effective data are unique and require distinct criteria for evaluation.
We argue that the most important criteria for identifying empirically sound qualitative research will reflect matters of execution (i.e., what happens in the field) rather than matters of conception (i.e., what is planned in the office ahead of time). Building on these arguments, we offer a set of five indicators and one precondition to identify quality of craft in qualitative social science—specifically, ethnographic and interview-based research that aims to contribute to cumulative knowledge by uncovering fundamental social facts about people, places, or phenomena.
We begin with the precondition of exposure, arguing that fieldworkers who have more direct contact with the social world and the people in it will generally come to produce greater qualitative insight. High levels of exposure can be achieved in multiple ways, and a study that follows a small group of people in one setting over a long period of time could ultimately achieve a similar or even greater level of exposure (i.e., more hours of contact) than a study involving a large sample size or multiple field sites.
Exposure alone, however, does not guarantee that the data produced will be high in quality. So, we also outline five indicators that, in the context of high levels of exposure, signal the kind of craft necessary produce qualitative insights that are empirically sound. The first of these is cognitive empathy—the degree to which the researcher came to understand the perceptions, meanings, and motivations of the people observed close to how they understand those aspects of themselves. The second is heterogeneity—the extent to which people or places are represented as diverse. We do not mean it necessarily includes people from multiple social groups or multiple settings; instead, that it reflects the ambiguities and contradictions that any fieldworker with sufficient exposure is likely to encounter. Our third criteria is palpability—the degree to which the evidence presented is not abstract but concrete. Palpable evidence involves not people’s generalizations about their experiences but rather specific instances involving specific people in specific settings at specific moments in time. Producing that kind of palpable data often involves our fourth indicator, follow-up—the extent to which the researcher collected data to answer questions that arose during the data collection process itself, such as by asking interview participants to elaborate or gathering additional data to clarify something unexpected that occurred in the field. Of course, what a particular fieldworker finds surprising will depend on that researcher’s background knowledge and experience, which brings us to our fifth indicator, self-awareness—the extent to which the researcher understands the impact of their presence and assumptions on those interviewed or observed.
These indicators are not exhaustive. Nor do we believe that all good fieldwork must exhibit all of these to the same degree. Yet these are common and important indicators of quality in ethnographic and interview-based research. We offer these indicators to inspire a discussion of qualitative literacy, with the goal of expanding it not only within academia but also among other readers who stand to benefit from the insights of qualitative research.
This post is part of our #ASA2022 blog series. Click here to order an exam copy.