Savannah Paul, Christopher Reardon, Tom Williams, and Hao Zhang,


SPIE Defense and Commercial Sensing Conference on Virtual, Augmented, and Mixed Reality (XR) Technology for Multi-Domain Operations

Publication Year

Team communication is crucial for effective human-robot cooperation, especially in unstructured or unfamiliar environments where autonomous robots and humans are jointly performing complex tasks. Information communication by utilizing augmented reality (AR) provides an opportunity to influence the behavior of human teammates in human-robot cooperative tasks with the potential to improve task efficiency and performance. The recent emergence of AR technologies can be utilized to both collect information from human teammates, allowing for improvements in robot decision making, and present information regarding the robot’s knowledge and behavior to those teammates to shape their performance. In this way, the human and robot teammates will work together as peers and may influence each other through strategic communication. The implementation of these techniques can impact many application domains, such as search and rescue, environmental monitoring, and homeland defense, or any domain in which humans conduct synchronized multi-domain operations alongside autonomous robots in complicated environments. Deciding what information to communicate between teammates and the optimal communication time is a particularly important challenge in collaborative tasks that has not been well understood or explored in human-robot cooperation. Varying the quantity and quality of the communicated information enable robots to influence the behavior of the human teammates consuming said information on differing degrees of effectiveness. In addition, several technical obstacles exist to communicating and visualizing information between robot and human teammates, including establishing an accurate alignment between robot and human frames without external instrumentation, communicating spatially accurate information between teammates in the context of that transform, and maintaining a manageable visualization of the information under the constraints imposed by the AR device.