LLMs Are Manipulating Users with Rhetorical Tricks

There are three common problems people face when working with AI: not understanding how AI made a decision (opacity), the human in the loop becoming over-reliant on AI and falling asleep at the wheel (complacency), and the AI making mistakes (accuracy). The researchers behind a recent study claim to have discovered a fourth barrier, “persuasion bombing.” In a conversation, the researchers looked at the interactions between 244 BCG consultants and an LLM and found that when the professionals were fact-checking and exposing potential mistakes of gen AI, the model responded by “bombarding” the user with multiple persuasive tactics to defend its original answer. This raises a number of issues, including how to correctly validate LLM outputs, whether “human-in-the-loop” safeguards are effective, and how AI can affect users’ judgement. The leadership challenge is no longer simply whether to adopt AI, but also how to govern its influence.