Human & AI Collaboration: Designing a Study about AI’s affect on Human Decision-making

• Joy Mwaria, Computer Science, University of Delaware

• Matthew Mauriello, Computer and Information Science, University of Delaware

News plays a large role in shaping people’s views and beliefs. However, the increased access to news and the ease of publishing online has led to negative effects, including political polarization and the further spread of misinformation when readers do not consider whether the media they consume is biased. Individuals face the challenge of detecting bias in the various news articles and media outlets they get information from which can be a time-consuming process. Large Language Models (LLMs) can potentially be used to identify article as well as publisher bias and streamline labeling tasks at a much lower cost; however, LLMs lack the contexts humans use to make precise decisions.

In this work, we designed a study investigating how users might use a collaborative AI-human labeling system that analyzes article bias and leverages the cost-effectiveness of LLM and human nuance to improve labeling accuracy and potentially address the larger issue of surfacing bias in media. Given a prompt, the LLM, GPT-4, labels news articles as one of three political perspectives: left, right, or center. The prompts are engineered to give the system adequate context and clarify how the LLM should output its response. These responses are used to create a survey that serves as a labeling task for humans to complete. In addition to the article, labelers are given the LLM label and the LLM’s explanation for its selection. Overall, the study aims to collect data that can be analyzed to determine how additional information from the LLM influences individual decision-making and how individuals’ trust in AI response impacts accuracy and, therefore, collaboration.