Meeting the machines half-way: testing LLMs for qualitative coding of values in texts
Which qualitative research tasks can LLMs supplement or replace? What are their strengths and limitations? How can human researchers collaborate with them?
Background:
An unprecedented amount of data is now available to humanities researchers, including parliamentary speeches, fairytales from across the globe, and historical newspapers. Large language models (LLMs) can speed up the analysis of this data by helping to summarise and automatically label text with topics or other categorical labels. Some claim that LLMs will replace human qualitative data coders altogether.
However, history teaches us that automation supplements and reconfigures jobs rather than replacing them. In relation to qualitative data analysis, it is still unclear how well LLMs can handle tasks that depend on deeper analysis and interpretation, such as value detection. Which qualitative research tasks can LLMs supplement or replace? What are their strengths and limitations? How can human researchers collaborate with them?
Researchers Dr Ajda Pretnar Žagar and Dr David Moats conducted an experiment to find out. They asked LLMs to code statements from parliamentary speeches by inferring which values they express. They then asked human qualitative researchers to identify values in the same set of texts and to review the output from humans and machines, deciding which they preferred.
Both humans and AI struggled with the value annotation task, albeit for different reasons.
LLMs used values not included on the list of values provided by the researchers and occasionally produced completely unsolicited, elaborate written justifications for each decision. These digressions from the task, or instances of ignoring instructions, make LLMs “less like a reliable automated process and more like an unruly research assistant who requires constant monitoring” (p. 6).
Assigning predetermined values to short texts from parliamentary speeches was also difficult for humans. First, human annotators often disagreed with each other about which values were present. To understand why, researchers conducted follow-up interviews about how the annotators made their decisions. The interviews showed that people relied on subtle language cues, cultural knowledge, and personal assumptions about what values are and how the task should be completed. AI models lacked this shared background.
The experiment suggests that LLMs might not be best suited to tasks that require a high level of precision. Instead, the researchers recommend drawing on LLMs for: 1) generating explanations, 2) retrieving examples from data, and 3) assisting with codebook development.
Read the full article at this link.