“Stealing is wrong… and too hard to pull off successfully”. Inclusion of Moral versus Capacity Information in Robot Command Rejection

Alyssa Hanson and Gordon Briggs and Ruchen Wen and Yifei Zhu and Tom Williams

ACM/AAAI Conference on AI, Ethics, and Society

In this work, we consider the inclusion of normative vs non-normative information in norm-violation responses. In particular, we consider situations where robots are given requests that simultaneously violate both normative constraints and non-normative capacity constraints. To understand what reasons robots should provide when confronted with such violations, we present the results of a human subjects experiment in which we systematically varied the degrees of both norm and capacity violation in human commands made toward robots, and then measured the effects of this variation on participants' preferred robot response choices. Our results (1) suggest that robots should consistently include normative information when rejecting a command, and (2) shed light on the contexts in which capacity information should also be included.