Integrity Based Explanations for Fostering Appropriate Trust in AI Agents

Journal article

Siddharth Mehrotra, Carolina Centeio Jorge, Catholijn M. Jonker, Myrthe L. Tielman
ACM Transactions on Interactive Intelligent Systems, Association for Computing Machinery, New York, NY, USA, 2023 Jul

DOI: 10.1145/3610578

Full-text

Cite

APA Click to copy
Mehrotra, S., Jorge, C. C., Jonker, C. M., & Tielman, M. L. (2023). Integrity Based Explanations for Fostering Appropriate Trust in AI Agents. ACM Transactions on Interactive Intelligent Systems. https://doi.org/10.1145/3610578

Chicago/Turabian Click to copy
Mehrotra, Siddharth, Carolina Centeio Jorge, Catholijn M. Jonker, and Myrthe L. Tielman. “Integrity Based Explanations for Fostering Appropriate Trust in AI Agents.” ACM Transactions on Interactive Intelligent Systems (July 2023).

MLA Click to copy
Mehrotra, Siddharth, et al. “Integrity Based Explanations for Fostering Appropriate Trust in AI Agents.” ACM Transactions on Interactive Intelligent Systems, Association for Computing Machinery, July 2023, doi:10.1145/3610578.

BibTeX Click to copy

@article{mehrotra2023a,
  title = {Integrity Based Explanations for Fostering Appropriate Trust in AI Agents},
  year = {2023},
  month = jul,
  address = {New York, NY, USA},
  journal = {ACM Transactions on Interactive Intelligent Systems},
  publisher = {Association for Computing Machinery},
  doi = {10.1145/3610578},
  author = {Mehrotra, Siddharth and Jorge, Carolina Centeio and Jonker, Catholijn M. and Tielman, Myrthe L.},
  month_numeric = {7}
}

Abstract:
Appropriate trust is an important component of the interaction between people and AI systems, in that ‘inappropriate’ trust can cause disuse, misuse or abuse of AI. To foster appropriate trust in AI, we need to understand how AI systems can elicit appropriate levels of trust from their users. Out of the aspects that influence trust, this paper focuses on the effect of showing integrity. In particular, this paper presents a study of how different integrity-based explanations made by an AI agent affect the appropriateness of trust of a human in that agent. To explore this, (1) we provide a formal definition to measure appropriate trust, (2) present a between-subject user study with 160 participants who collaborated with an AI agent in such a task. In the study, the AI agent assisted its human partner in estimating calories on a food plate by expressing its integrity through explanations focusing on either honesty, transparency or fairness. Our results show that (a) an agent who displays its integrity by being explicit about potential biases in data or algorithms achieved appropriate trust more often compared to being honest about capability or transparent about the decision-making process, and (b) subjective trust builds up and recovers better with honesty-like integrity explanations. Our results contribute to the design of agent-based AI systems that guide humans to appropriately trust them, a formal method to measure appropriate trust, and how to support humans in calibrating their trust in AI.