Google Scholar
Amazing news!👏🏼 Our paper, The Persuasive Power of Large Language Models, got accepted to ICWSM 2024! 🎉 This means that we will be presenting the work at the conference in Buffalo in June.
The majority of the great work was done by Simon Martin Breum, Daniel Vædele Egdal, and Victor Gram Mortensen, data science MSc students at ITU.
🤔 What is it about?
The paper dives into the broad narrative of applications of large language models (LLMs) and their abilities in persuasive communication. We explore how LLM agents can mimic a simple dyadic conversation in which one of the agents trying to convince the other. The main questions in the paper are:
Can LLMs emulate realistic dynamics of persuasion and opinion change?
Can LLMs be prompted to generate arguments using various persuasion strategies?
Are arguments thate are persuasive to LLM agents also perceived as effective by humans?
We apply different personas to the convincing agents which alter their persuasive strategies. We also explore how we can control the scepticism of an agent, and how succeptiple it is to change its opinion on the timely topic of climate change.
The image above illustrates our experimental setup.
📊 How well did that go?
With the experimental setup, we can particularly show two points:
🎭 We can alter persuasive strategies using personas: In the work of Monti et al. (2022), language of opinion change was studied and certain social personalities were found to be more effective in inducing change of opinion. Using only LLMs, our study partly confirms the findings showing that inducing communicative strategies can improve persuasive capabilities.
🤔 Skepticism is a controlable variable: Our work shows that we can control the level of skepticism of an agents - this is consistent across all convinver personalities.
🙋🏼 What about human evaluation?
Using MTurk, we evaluated a subset of the persuasive arguments in a pairwise fashion. Here the annotators should select which of two arguments were deemed the most persuasive. With these human preferences, we could use the Bradley-Terry model to estimate the strengths of the different communicative strategies, as well as probabilities of one strategy winning over the other.
🤷🏼♂️ Why is this important?
With the easy and widespread access to LLMs capable of quickly generating content, we can anticipate a flood of AI-generated content on social media platforms. Understanding their behavior and the public’s reaction to such content is crucial for identifying potential risks. A recent pre-print by Salvi et al., (2024): “On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial”, interestingly demonstrates the impact of personalization in AI-human dialogue settings. These studies hold significance not only for the general public but also for policymakers and social media platforms.
🤙🏼 Want to know more?
For a more detailed walkthrough of the paper, please see this blog post or check out the paper.