Qualitative research methods are data, time and manpower intensive. The use of Artificial Intelligence (AI), such as Large Language Models (LLMs), has been proven effective in a variety of text-based quantitative tasks, but their use in qualitative research remains underexplored. We investigated the effectiveness of an LLM-based workflow to conduct a thematic analysis on transcripts from 12 focus groups (N=19 children 9-15-years-old, N=19 adults 38-56-years old) and compared the LLM-derived (sub-)themes to those of a researcher-led thematic analysis. Using a GPT-3.5-turbo-instruct model with model temperature set to 0, we developed an LLM-based workflow to (1) annotate the data line-by-line, (2) summarise the annotations into a list of sub-themes and (3) iteratively group the sub-themes into higher-level themes. The LLM-based workflow generated 13 themes which covered all 6 researcher-generated themes. When comparing sub-themes, we noticed that the LLM-based workflow did not detect 4 of the sub-themes identified during the research-led analysis. When exploring this finding further, we theorise that this is due to the specific topic expertise that the researchers had on the studied topic. Overall, our findings suggest that a ‘researcher-AI collaborative system’ may be best suited to analyse qualitative data. LLM-based workflows can annotate data and generate (sub-)themes to support researchers to familiarise themselves with the data. LLM-generated (sub-)themes can then be further refined with input from researchers to provide topic expertise. This researcher-AI collaborative system combines the resource-effectiveness of AI tools with the contextual knowledge and domain expertise of researchers.

Authors:
Aimee Pink, Agency for Science, Technology and Research (A*STAR), Singapore
Prasanta Bhattacharya, Agency for Science, Technology and Research, Singapore
Siyuan Brandon Loh, Agency for Science, Technology and Research (A*STAR), Singapore
Hui Si Oh, Agency for Science, Technology and Research (A*STAR), Singapore
Siti Amelia Juraimi, Agency for Science, Technology and Research (A*STAR), Singapore
Nandini Anant, Agency for Science, Technology and Research (A*STAR), Singapore
Mary F-F. Chong, National University Singapore, Singapore
Anna Fogel, Agency for Science, Technology and Research (A*STAR), Singapore
Florence Sheen, Loughborough University, United Kingdom
Wong Liang Ze, Agency for Science, Technology and Research (A*STAR), Singapore

About the Presenter(s)
Dr Aimee Pink is a Senior Scientist at the Agency for Science, Technology and Research (A*STAR), Singapore. Her current work focuses on understanding consumer perceptions of novel foods, particularly focusing on children.

Connect on Linkedin
https://www.linkedin.com/in/aimee-pink/

See this presentation on the full schedule – Monday Schedule

Bookmark