The Conversational Action Test: detecting the artificial sociality of AI

Drawing on the ‘Voigt-Kampff Empathy Test’—Phillip K. Dick’s fictionalized version of Turing’s famous thought experiment—we propose a Conversational Action Test (CAT) to identify and evaluate conversational AI voice agents. We compare social actions in a range of telephone service encounters where one party is an artificial conversational agent to a range of similar human-human calls. We focus on the situated interactional practices through which the agent ‘passes’ for human, and how this reveals the limits of behavioral testing for AI in task-based routine service interactions. We discuss the implications of the CAT for the design and evaluation of conversational AI, and for the notion of ‘humanness’ as a benchmark. Data include publicly available human/AI service calls and comparable human-human calls in British and American English.