Employees Aren’t Questioning AI Advice Enough

Many believe that making AI’s decision-making more transparent will help users of AI tools overcome potential biases in the tools’ thinking. But a new study found that people often avoid finding out “why” an AI made a decision when it threatens their comfort or incentives. In an online experiment with 2,512 participants acting as loan officers on real $10,000 requests, most sought AI’s assistance to determine applicants’ risk of defaulting on the loan, but fewer than half chose to view explanations about how risk assessments were determined. This was especially true when bonuses for participants depended on repayment or when they were told that explanations may include potential race- or gender-based bias. Explanations, when viewed, increased overrides of the AI. To ensure responsible AI use, mere availability of explanation is not enough. Companies must create incentives, regulation, and organizational design that underscore and affirm human judgement and review.