HRI ’24: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

HRI ’24: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

Full Citation in the ACM Digital Library

SESSION: Keynote Talks

Creating Expressive and Engaging Robotic Characters

  • James Kennedy

Designing robots for entertainment scenarios presents many of the same challenges encountered in other Human-Robot Interaction (HRI) deployments, such as maintaining relationships over time, operating in unpredictable and noisy environments, and personalizing interactions for diverse users and user groups. However, the success of such interactions also hinges on additional factors such as personality, expressivity, and storytelling. Whether creating a new robotic character, or bringing an existing character to life, a great deal of both technical and artistic collaboration is required to create compelling experiences. In this talk, I will showcase several robotic character projects that are at various stages of real-world deployment and testing. Using these projects as examples, I will discuss the process of going from a research prototype to creating believable, expressive and engaging characters. This will include dynamic robots that are designed for large audiences, where nonverbal behavior is crucial, and robots that interact with smaller groups, relying more heavily on conversation and language understanding.

What Do Foundation Models have to Do With and For HRI?

  • Carolina Parada

Foundation models have unlocked major advancements in AI. What do foundation models have to do with and for Human-Robot Interaction – And how can HRI help unlock more powerful foundation models for robot learning and embodied reasoning? In this talk, I will discuss examples of how foundation models could enable a step function in human-robot interaction research, including: how to leverage foundation models to enable multimodal human-robot communication, enable non-expert users to teach robots new low-level skills and personalized high level plans through natural interactions, create new expressive robot behaviors, etc. At the same time, foundation models still have significant gaps in human-robot interaction contexts. I will share early insights showing that HRI could be key to evolving the foundation models themselves, enabling even more powerful interactions, and improving robot learning. The fields of HRI and robot learning seem to have evolved and grown in parallel, but foundation models might be the breakthrough we needed to bring these fields together. Now there is a unique opportunity for HRI to unlock robot learning in-the-wild, not only because it will yield robots that are more useful and adaptable to humans, but because it will enable improving the foundation models that will likely affect every aspect of robot learning.

Social-Digital Vulnerability

  • Ryan Calo

This talk describes the phenomenon of socio-digital vulnerability (SDV). SDV refers to the susceptibility of individuals and groups within mediated environments to decisional, social, or constitutive interference. Drawing from work in law and design, Professor Calo uses dark patterns, robots, generative artificial intelligence, and other examples to evidence he problem of SDV; he argues that vulnerability in mediated environments is best under in context, rather than as a binary; and he suggests policy frameworks that go behind harm mitigation to address the power imbalances that underpin SDV.

SESSION: Full Research Papers

Intentional User Adaptation to Shared Control Assistance

  • Reuben M. Aronson
  • Elaine Schaertl Short

Shared control approaches to robot assistance, which predict a user’s goal based on their control input and provide autonomous assistance towards the predicted goal, typically assume that user behavior remains the same despite the presence of the assistance and rely on this assumption to infer user goals. However, people operating assisted systems continuously observe the robot behaving differently from their expectations, which may lead them to adapt their control behavior to better achieve their desired outcomes. In this paper, we show that users both change their control behavior when assistance is added and describe these changes as responses to the new system dynamics. In a computer-based bubble popping study, participants report changing their strategies with different levels of assistance, and analysis of their actual control input validates this change. In an in-the-wild robot study, participants teleoperated a robot to pick up a cup despite the presence of “assistance” that drives the system away from the true goals of the task. Participants can overcome the “assistance” and reach the goal, which requires them to correct for the novel system dynamics. These results motivate further research in user-centered design and evaluation of assistive systems that treat the user as intentional.

“Oh, Sorry, I Think I Interrupted You”: Designing Repair Strategies for Robotic Longitudinal Well-being Coaching

  • Minja Axelsson
  • Micol Spitale
  • Hatice Gunes

Robotic well-being coaches have been shown to successfully promote people’s mental well-being. To provide successful coaching, a robotic coach should have the capability to repair the mistakes it makes. Past investigations of robot mistakes are limited to game or task-based, one-off and in-lab studies. This paper presents a 4-phase design process to design repair strategies for robotic longitudinal well-being coaching with the involvement of real-world stakeholders: 1) designing repair strategies with a professional well-being coach; 2) a longitudinal study with the involvement of experienced users (i.e., who had already interacted with a robotic coach) to investigate the repair strategies defined in (1); 3) a design workshop with users from the study in (2) to gather their perspectives on the robotic coach’s repair strategies; 4) discussing the results obtained in (2) and (3) with the mental well-being professional to reflect on how to design repair strategies for robotic coaching. Our results show that users have different expectations for a robotic coach than a human coach, which influences how repair strategies should be designed. We show that different repair strategies (e.g., apologizing, explaining, or repairing empathically) are appropriate in different scenarios, and that preferences for repair strategies change during longitudinal interactions with the robotic coach.

User-Designed Human-UAV Interaction in a Social Indoor Environment

  • Alisha Bevins
  • Siya Kunde
  • Brittany A. Duncan

The purpose of this project is to understand how people would expect to interact with an Unmanned Aerial Vehicle (UAV) in a social indoor environment under friendly, neutral, or adversarial contexts. The three environments will include one setting with the UAV serving as a tour guide, one as a security guard, and one as a food delivery mechanism. This work is novel in its inquiry into the affective nature of the interaction, comparison across situational contexts, and ability to compare preferences both within and between participants.Our findings will help researchers plan for appropriate safety and comfort measures, while being cognizant of the participants’ preferences for and understanding of how drones operate. This study examines realistic indoor scenarios for which each participant designs their preferred interaction and presents exploratory results, including comparison to prior work with respect to motion gestures and comfortable approach distances. Initial findings suggest the importance of visibility of approaches, selecting approach heights relative to the person and based on the context of interaction, and criticality of the initial direction of motion when classifying the communicative content of UAV flight paths.

Evaluating the Impact of Personalized Value Alignment in Human-Robot Interaction: Insights into Trust and Team Performance Outcomes

  • Shreyas Bhat
  • Joseph B. Lyons
  • Cong Shi
  • X. Jessie Yang

This paper examines the effect of real-time, personalized alignment of a robot’s reward function to the human’s values on trust and team performance. We present and compare three distinct robot interaction strategies: a non-learner strategy where the robot presumes the human’s reward function mirrors its own; a non-adaptive-learner strategy in which the robot learns the human’s reward function for trust estimation and human behavior modeling, but still optimizes its own reward function; and an adaptive-learner strategy in which the robot learns the human’s reward function and adopts it as its own. Two human-subject experiments with a total number of N=54 participants were conducted. In both experiments, the human-robot team searches for potential threats in a town. The team sequentially goes through search sites to look for threats. We model the interaction between the human and the robot as a trust-aware Markov Decision Process (trust-aware MDP) and use Bayesian Inverse Reinforcement Learning (IRL) to estimate the reward weights of the human as they interact with the robot. In Experiment 1, we start our learning algorithm with an informed prior of the human’s values/goals. In Experiment 2, we start the learning algorithm with an uninformed prior. Results indicate that when starting with a good informed prior, personalized value alignment does not seem to benefit trust or team performance. On the other hand, when an informed prior is unavailable, alignment to the human’s values leads to high trust and higher perceived performance while maintaining the same objective team performance.

Aligning Human and Robot Representations

  • Andreea Bobu
  • Andi Peng
  • Pulkit Agrawal
  • Julie A Shah
  • Anca D. Dragan

To act in the world, robots rely on a representation of salient task aspects: for example, to carry a coffee mug, a robot may consider movement efficiency or mug orientation in its behavior. However, if we want robots to act for and with people, their representations must not be just functional but also reflective of what humans care about, i.e. they must be aligned. We observe that current learning approaches suffer from representation misalignment, where the robot’s learned representation does not capture the human’s representation. We suggest that because humans are the ultimate evaluator of robot performance, we must explicitly focus our efforts on aligning learned representations with humans, in addition to learning the downstream task. We advocate that current representation learning approaches in robotics should be studied from the perspective of how well they accomplish the objective of representation alignment. We mathematically define the problem, identify its key desiderata, and situate current methods within this formalism. We conclude by suggesting future directions for exploring open challenges.

CARMEN: A Cognitively Assistive Robot for Personalized Neurorehabilitation at Home

  • Anya Bouzida
  • Alyssa Kubota
  • Dagoberto Cruz-Sandoval
  • Elizabeth W. Twamley
  • Laurel D. Riek

Cognitively assistive robots (CARs) have great potential to extend the reach of clinical interventions to the home. Due to the wide variety of cognitive abilities and rehabilitation goals, these systems must be flexible to support rapid and accurate implementation of intervention content that is grounded in existing clinical practice. To this end, we detail the system architecture of CARMEN (Cognitively Assistive Robot for Motivation and Neurorehabilitation), a flexible robot system we developed in collaboration with our key stakeholders: clinicians and people with mild cognitive impairment (PwMCI). We implemented a well-validated compensatory cognitive training (CCT) intervention on CARMEN, which it autonomously delivers to PwMCI. We deployed CARMEN in the homes of these stakeholders to evaluate and gain initial feedback on the system. We found that CARMEN gave participants confidence to use cognitive strategies in their everyday life, and participants saw opportunities for CARMEN to exhibit greater levels of autonomy or be used for other applications. Furthermore, elements of CARMEN are open source to support flexible home-deployed robots. Thus, CARMEN will enable the HRI community to deploy quality interventions to robots, ultimately increasing their accessibility and extensibility.

Mind-Body-Identity: A Scoping Review of Multi-Embodiment

  • Karla Bransky
  • Penny Sweetser
  • Sabrina Caldwell
  • Kingsley Fletcher

Multi-embodied agents can have both physical and virtual bodies, moving between real and virtual environments to meet user needs, embodying robots or virtual agents alike to support extended human-agent relationships. As a design paradigm, multi-embodiment offers potential benefits to improve communication and access to artificial agents, but there are still many unknowns in how to design these kinds of systems. This paper presents the results of a scoping review of the multi-embodiment research, aimed at consolidating the existing evidence and identifying knowledge gaps. Based on our review, we identify key research themes of: multi-embodied systems, identity design, human-agent interaction, environment and context, trust, and information and control. We also identify 16 key research challenges and 12 opportunities for future research.

Toward Family-Robot Interactions: A Family-Centered Framework in HRI

  • Bengisu Cagiltay
  • Bilge Mutlu

As robotic products become more integrated into daily life, there is a greater need to understand authentic and real-world human-robot interactions to inform product design. Across many domestic, educational, and public settings, robots interact with not only individuals and groups of users, but also families, including children, parents, relatives, and even pets. However, products developed to date and research in human-robot and child-robot interactions have focused on the interaction with their primary users, neglecting the complex and multifaceted interactions between family members and with the robot. There is a significant gap in knowledge, methods, and theories for how to design robots to support these interactions. To inform the design of robots that can support and enhance family life, this paper provides (1) a narrative review exemplifying the research gap and opportunities for family-robot interactions and (2) an actionable family-centered framework for research and practices in human-robot and child-robot interaction.

A Social Approach for Autonomous Vehicles: A Robotic Object to Enhance Passengers’ Sense of Safety and Trust

  • Srivatsan Chakravarthi Kumaran
  • Toam Bechor
  • Hadas Erel

One of the central challenges in designing autonomous vehicles concerns passenger trust and sense of safety. This challenge is related to passengers’ well-established past experience with non-autonomous vehicles, which leads to concern about the absence of a driver. We explored whether it is possible to address this challenge by designing an interaction with a simple robotic object positioned on the vehicle’s dashboard. The robot greeted the passenger, indicated that the vehicle was attentive to its surroundings, and informed the passenger that the drive was about to begin. We evaluated whether the robot’s non-verbal behavior would provide the signals and social experience required to support passengers’ trust and sense of safety. In an in-person (in-situ) experiment, participants were asked to enter an autonomous vehicle and decide if they were willing to go for a drive. As they entered the vehicle, the robot performed the designed behaviors. We evaluated the participants’ considerations and experience while they made their decision. Our findings indicated that participants’ trust ratings and safety-related experience were higher than those of a baseline group who did not interact with the robot. Participants also perceived the robot as providing companionship during a lonely experience. We suggest that robotic objects are a promising technology for enhancing passengers’ experience in autonomous vehicles.

“Sorry to Keep You Waiting”: Recovering from Negative Consequences Resulting from Service Robot Unintended Rejection

  • Xiaoyu Chang
  • Yanheng Li
  • Sijia Liu
  • Ling Ma
  • Ray Lc

Robots are increasingly deployed in crowded, large-scale environments where the demands on their services can outweigh their ability to respond. When robots fail to respond, humans may interpret the unintended consequence negatively as a form of rejection, leading to a loss of trust. How do service robots recover from such rejection to remediate human trust due to perceived rejection? We created a task mimicking shopping malls where the robot arm is asked to provide coffee, juice, or tea to participants. When the robot rendered service elsewhere, participants reported feeling excluded and less trusting of the robot. When the robot subsequently apologized or provided promise of future favor, participants regained trust in the robot, with favor rendering yielding significantly more trust responses. This study highlights the importance of understanding inadvertently negative consequences of robot behaviors, and suggests design solutions for overcoming this negative perception through remediation strategies.

Investigating the Impact of Gender Stereotypes in Authority on Avatar Robots

  • Yuan-Chia Chang
  • Daniel J. Rea
  • Takayuki Kanda

We investigate how gender stereotypes in authority influence the perceptions and behavior of avatar robots operators and their interlocutors. Gender stereotypes, which typically place men in more authoritative positions than women, are present in not only inter-human but also human-robot interaction. As avatar robots become more integrated into our lives and serve for diverse usages, they may be utilized in positions where they require authority. We study how avatar robot gender and operator gender affect expressions and perception of gender stereotypes in a customer service scenario with 41 pairs of participants. Operators controlled binary gendered avatar robots one at a time, acting as shopkeepers that had to assert authority over customers behaving improperly. The operators perceived their authority to be higher with male avatar robots compared to female ones, regardless of operator gender. We did not detect an effect on customer’s perception of the shopkeeper’s authority. While less than half of operators and customers perceived authority for reasons related to traditional gender stereotypes, others observed behaviors that did not align with stereotypes. Avatar embodiment may also help operators assert authority safely due to being physically hidden from the customers.

Integrating Flow Theory and Adaptive Robot Roles: A Conceptual Model of Dynamic Robot Role Adaptation for the Enhanced Flow Experience in Long-term Multi-person Human-Robot Interactions

  • Huili Chen
  • Sharifa Alghowinem
  • Cynthia Breazeal
  • Hae Won Park

In this paper, we introduce a novel conceptual model for a robot’s behavioral adaptation in its long-term interaction with humans, integrating dynamic robot role adaptation with principles of flow experience from psychology. This conceptualization introduces a hierarchical interaction objective grounded in the flow experience, serving as the overarching adaptation goal for the robot. This objective intertwines both cognitive and affective sub-objectives and incorporates individual and group-level human factors. The dynamic role adaptation approach is a cornerstone of our model, highlighting the robot’s ability to fluidly adapt its support roles-from leader to follower-with the aim of maintaining equilibrium between activity challenge and user skill, thereby fostering the user’s optimal flow experiences. Moreover, this work delves into a comprehensive exploration of the limitations and potential applications of our proposed conceptualization. Our model places a particular emphasis on the multi-person HRI paradigm, a dimension of HRI that is both under-explored and challenging. In doing so, we aspire to extend the applicability and relevance of our conceptualization within the HRI field, contributing to the future development of adaptive social robots capable of sustaining long-term interactions with humans.

Interactive Human-Robot Teaching Recovers and Builds Trust, Even With Imperfect Learners

  • Vivienne Bihe Chi
  • Bertram F. Malle

Building and maintaining trust is critically important for continued human-robot teaching and the prospect of robots learning social skills from natural environments. Whereas previous work often explored strategies to reduce system errors, mitigate trust loss, or enhance learning by interactive teaching, few studies have investigated the possible benefits of fully engaged, interactive teaching on human trust. Motivated by a pair of discrepant previous investigations, the present studies for the first time directly tested the causal impact of interactivity on the loss and recovery of trust in a human-robot social skills training context. Building on a previously developed experimental paradigm, we randomly assigned participants to one of two modes of interaction: interactive teacher vs. supervisor of an experimentally controlled virtual robot. The robot was engaged in learning norm-appropriate behavior in a healthcare setting and improved from mistake-prone to near-flawless performance. Participants indicated their changing trust during the 15-trial training session and how much they attributed the robot’s improvement to their own training contributions. Interactive teachers were more resilient to initial trust loss, showed increased trust in the robot’s performance on additional tasks, and attributed more of the robot’s improvement to themselves than did supervisors, even when the robots were slow learners.

Bridging HRI Theory and Practice: Design Guidelines for Robot Communication in Dairy Farming

  • Nazli Cila
  • Irene González González
  • Jan Jacobs
  • Marco Rozendaal

Using HRI theory to inform robot development is an important, but difficult, endeavor. This paper explores the relationship between HRI theory and HRI practice through a design project on the development of design guidelines for human-robot communication together with a dairy farming robot manufacturer. The design guidelines, a type of intermediate-level knowledge, were intended to enrich the specialized knowledge of the company on farming context with relevant academic knowledge. In this process, we identified that HRI theories were used as a frame, a tool, best practices, and a reference; while the HRI practice provided a context, a reference, and validation for the theories. Our intended contribution is to propose a means to facilitate exchanges both ways between HRI theory and practice and add to the emerging repertoire of designerly ways of producing knowledge in HRI.

“An Emotional Support Animal, Without the Animal”: Design Guidelines for a Social Robot to Address Symptoms of Depression

  • Sawyer Collins
  • Kenna Baugus Henkel
  • Zachary Henkel
  • Casey C. Bennett
  • Cedomir Stanojevic
  • Jennifer A. Piatt
  • Cindy L. Bethel
  • Selma Sabanović

Socially assistive robots can be used as therapeutic technologies to address depression symptoms. Through three sets of workshops with individuals living with depression and clinicians, we developed design guidelines for a personalized therapeutic robot for adults living with depression. Building on the design of Therabot, workshop participants discussed various aspects of the robot’s design, sensors, behaviors, and a robot connected mobile phone app. Similarities among participants and workshops included a preference for a soft textured exterior and natural colors and sounds. There were also differences – clinicians wanted the robot to be able to call for aid, while participants with depression differed in their degree of comfort in sharing data collected by the robot with clinicians.

The Effects of Observing Robotic Ostracism on Children’s Prosociality and Basic Needs

  • Filipa Correia
  • Isabel Neto
  • Soraia Paulo
  • Patricia Piedade
  • Hadas Erel
  • Ana Paiva
  • Hugo Nicolau

Research on robotic ostracism is still scarce and has only explored its effects on adult populations. Although the results revealed important carryover effects of robotic exclusion, there is no evidence yet that those results occur in child-robot interactions. This paper provides the first exploration of robotic ostracism with children. We conducted a study using the Robotic Cyberball Paradigm in a third-person perspective with a sample of 52 children aged between five to ten years old. The experimental study had two conditions: Exclusion and Inclusion. In the Exclusion condition, children observed a peer being excluded by two robots; while in the Inclusion condition, the observed peer interacted equally with the robots. Notably, even 5-year-old children could discern when robots excluded another child. Children who observed exclusion reported lower levels of belonging and control, and exhibited higher prosocial behaviour than those witnessing inclusion. However, no differences were found in children’s meaningful existence, self-esteem, and physical proximity across conditions. Our user study provides important methodological considerations for applying the Robotic Cyberball Paradigm with children. The results extend previous literature on both robotic ostracism with adults and interpersonal ostracism with children. We finish discussing the broader implications of children observing ostracism in human-robot interactions.

A Virtual Reality Framework for Human-Driver Interaction Research: Safe and Cost-Effective Data Collection

  • Luca Crosato
  • Chongfeng Wei
  • Edmond S. L. Ho
  • Hubert P. H. Shum
  • Yuzhu Sun

The advancement of automated driving technology has led to new challenges in the interaction between automated vehicles and human road users. However, there is currently no complete theory that explains how human road users interact with vehicles, and studying them in real-world settings is often unsafe and time-consuming. This study proposes a 3D Virtual Reality (VR) framework for studying how pedestrians interact with human-driven vehicles. The framework uses VR technology to collect data in a safe and cost-effective way, and deep learning methods are used to predict pedestrian trajectories. Specifically, graph neural networks have been used to model pedestrian future trajectories and the probability of crossing the road. The results of this study show that the proposed framework can be for collecting high-quality data on pedestrian-vehicle interactions in a safe and efficient manner. The data can then be used to develop new theories of human-vehicle interaction and aid the Autonomous Vehicles research.

Follow Me: Anthropomorphic Appearance and Communication Impact Social Perception and Joint Navigation Behavior

  • Pia Dautzenberg
  • Stefan Ladwig
  • Astrid Marieke Rosenthal-von der Pütten

This study addresses how anthropomorphic features shape users’ social perception and trust towards service robots and whether anthropomorphic characteristics influence the way people jointly navigate with them facing several obstacles in a course. Therefore, an experimental study was conducted where two communication and appearance designs (humanlike vs. machinelike) were examined for a service robot that provides transportation of goods by semi-automated following. The results of the study indicate that the humanlike robot design is rated more competent, warmer, less discomforting, and is generally preferred. Furthermore, participants jointly navigating with the humanlike designed robot walked around obstacles significantly more often indicating a more considerate navigation behavior and better remembering of system limits; both probably evoked by the humanlike design characteristics. In sum, the results of this study provide intriguing implications on how to target HRI for the service robot examined to enhance pleasant and error-free interaction.

Can’t You See I Am Bothered? Human-inspired Suggestive Avoidance for Robots

  • Kanghui Du
  • Drazen Brsčić
  • Yuyi Liu
  • Takayuki Kanda

We studied how robots could stop people from repeatedly obstructing them by using reactions that people commonly use. From 35 hours of observation of people in a shopping mall, we identified one commonly used reaction, which we named suggestive avoidance. It consists of making a quick movement to the side while rotating the body and gaze toward the obstructing person, in a way that seems to imply that they were bothered by the obstruction. We modeled the human suggestive avoidance behavior, implemented it on a robot, and tested it both in a lab experiment and a field study. The results from the lab study confirmed that people perceive a robot using suggestive avoidance as being more bothered, as well as more human-like. The field study showed that when a robot uses suggestive avoidance people are less likely to bother it again.

Field Trial of an Autonomous Shopworker Robot that Aims to Provide Friendly Encouragement and Exert Social Pressure

  • Sachi Edirisinghe
  • Satoru Satake
  • Drazen Brscic
  • Yuyi Liu
  • Takayuki Kanda

We developed an autonomous hatshop robot for encouraging customers to try on hats by providing comments that appropriately fit their actions, and in such a way also indirectly exerting social pressure. To enable it to offer such a service smoothly in a real shop, we developed a large system (around 150k lines of code with 23 ROS packages) integrated with various technologies, like people tracking, shopping activity recognition and navigation. The robot needed to move in narrow corridors, detect customers, and recognise their shopping activities. We employed an iterative development process, repeating trial-and-error integration with the robot in the actual shop, while also collecting real-world data during field-testing. This process enabled us to improve our shopping activity recognition system by collecting real-world data, and to adapt our software modules to the target shop environment. We report the lessons learnt during our system development process. The results of our 11-day field trial show that our robot was able to provide its services reasonably well. Many customers expressed a positive impression of the robot and its services.

The Power of Opening Encounters in HRI: How Initial Robotic Behavior Shapes the Interaction that Follows

  • Hadas Erel
  • Agam Oberlender
  • Juna Khatib
  • Noam Freund
  • Omer Sadeh
  • Julian Waksberg
  • Elior Carsenti

Opening encounters are a fundamental component of every interaction. Psychology research highlights the valence of opening encounters as one of the main factors shaping the interaction that follows. We evaluated whether opening encounters would have a similarly powerful effect on human-robot interactions. Specifically, we tested how positive and negative opening encounters with a robot would impact the subsequent interaction. Participants interacted with a robot that performed gestures communicating different valences of opening encounters: Positive, Negative, or No opening encounter. To evaluate the impact on the subsequent interaction, we measured participants’ willingness to comply with a help request presented by the robot and their perception of the robot. Our results indicated that most participants in the Positive opening encounter condition helped the robot and described a positive overall perception. An opposite pattern emerged in the other two conditions. Almost none of the participants helped the robot, and the perception of the robot was less positive. Our findings suggest that opening encounters with robots should be carefully considered due to their impact on the interaction that follows.

I Need to Pass Through! Understandable Robot Behavior for Passing Interaction in Narrow Environment

  • Yusuke Fujioka
  • Yuyi Liu
  • Takayuki Kanda

We developed a motion control algorithm for a social mobile robot to intuitively convey its intent via social cues to pass through aisles and avoid misunderstanding in passing interactions with people, which frequently occur when a robot navigates in narrow shared environments. Inspired by observations of human behavior, the proposed algorithm estimates the extent to which a person understands the robot’s intent on the basis of the person’s reactions to the oncoming robot and provides the robot with corresponding motion strategies for effective passing interactions. We implemented the proposed algorithm onto an omni-directional humanoid robot and conducted a field study over six days in a store with 75 cm wide narrow aisles. The resulting behaviors of 50 customers demonstrated that our proposed method provided people with a clearer understanding of the robot’s intent in passing interactions, and thus the robot had more opportunity (73.1%) to pass through aisles compared to 16.7% if the robot moved and then waited for people to make space.

Effect of Social Robot’s Role and Behavior on Parent-Toddler Interaction

  • Omer Gvirsman
  • Goren Gordon

Social robots, designed to interact with people through natural communication modes like speech, body motion, gestures, and facial expressions, have been extensively studied in child-robot interaction for educational purposes. Recently, social robots have been explored in triadic parent-child-robot interactions, showing promise due to their interactivity, computational power, and physical presence, which enable multimodal natural communication and cater to toddlers’ developmental stages and physical curiosity. However, these have focused only on shared reading experiences and engaged older children, rather than toddlers. We developed two games, one with two levels of robot scaffolding, and another with either structured or unstructured design. We then explored, in two studies, how a social robot’s assigned role and behaviors influence the engagement of parents and toddlers with the robot and their interaction with each other. Our results show that parents affectively scaffolded their children less when the robot increased its scaffolding behaviors and that parents provided more scaffolding in a structured game with the robot, whereas in an unstructured game the dyad exhibited more cooperation in which children exhibited more independence. These findings can contribute to a better understanding of interaction design, triadic dynamics, and the role of the robot in parent-toddler-robot scenario.

A System for Human-Robot Teaming through End-User Programming and Shared Autonomy

  • Michael Hagenow
  • Emmanuel Senft
  • Robert Radwin
  • Michael Gleicher
  • Michael Zinn
  • Bilge Mutlu

Many industrial tasks-such as sanding, installing fasteners, and wire harnessing-are difficult to automate due to task complexity and variability. We instead investigate deploying robots in an assistive role for these tasks, where the robot assumes the physical task burden and the skilled worker provides both the high-level task planning and low-level feedback necessary to effectively complete the task. In this article, we describe the development of a system for flexible human-robot teaming that combines state-of-the-art methods in end-user programming and shared autonomy and its implementation in sanding applications. We demonstrate the use of the system in two types of sanding tasks, situated in aircraft manufacturing, that highlight two potential workflows within the human-robot teaming setup. We conclude by discussing challenges and opportunities in human-robot teaming identified during the development, application, and demonstration of our system.

The Power of Advice: Differential Blame for Human and Robot Advisors and Deciders in a Moral Advising Context

  • Alyssa Hanson
  • Nichole D. Starr
  • Cloe Emnett
  • Ruchen Wen
  • Bertram F. Malle
  • Tom Williams

Due to their unique persuasive power, language-capable robots must be able to both adhere to and communicate human moral norms. These requirements are complicated by the possibility that people may blame humans and robots differently for violating those norms. These complications raise particular challenges for robots giving moral advice to decision makers, as advisors and deciders may be blamed differently for endorsing the same moral action. In this work, we thus explore how people morally evaluate human and robot advisors to human and robot deciders. In Experiment 1 (n = 555), we examine human blame judgments of robot and human moral advisors and find clear evidence for an advice as decision hypothesis: advisors are blamed similarly to how they would be blamed for making the decisions they advised. In Experiment 2 (n = 1326), we examine blame judgments of a robot or human decider following the advice of a robot or human moral advisor. We replicate the results from Experiment 1 and also find clear evidence for a differential dismissal hypothesis: moral deciders are penalized for ignoring moral advice, especially when a robot ignores human advice. Our results raise novel questions about people’s perception of moral advice, especially when it involves robots, and present challenges for the design of morally competent robots.

Human perception of swarm fragmentation

  • Aymeric Hénard
  • Etienne Peillard
  • Jérémy Rivière
  • Sébastien Kubicki
  • Gilles Coppin

In the context of robot swarms, fragmentation refers to a breakdown in communication and coordination among the robots. This fragmentation can lead to issues in the swarm self-organisation, especially the loss of efficiency or an inability to perform their tasks. Human operators influencing the swarm could prevent fragmentation. To help them in this task, it is necessary to study the ability of humans to perceive and anticipate fragmentation. This article studies the perception of different types of fragmentation occurring in swarms depending on their behaviour selected amongst swarming, flocking, expansion and densification. Thus, we characterise human perception thanks to two metrics based on the distance separating fragmented groups and the separation speed. The experimentation protocol consists of a binary discrimination task in which participants have to assess the presence of fragmentation. The results show that detecting fragmentation for expansion behaviour and anticipating fragmentation, in general, are challenging. Moreover, they show that humans rely on separation distance and speed to infer the presence or absence of fragmentation. Our study paves the way for new research that will provide information to humans to better anticipate and efficiently prevent the occurrence of swarm fragmentation.

PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning

  • Simon Holk
  • Daniel Marta
  • Iolanda Leite

Preference-based reinforcement learning (RL) has emerged as a new field in robot learning, where humans play a pivotal role in shaping robot behavior by expressing preferences on different sequences of state-action pairs. However, formulating realistic policies for robots demands responses from humans to an extensive array of queries. In this work, we approach the sample-efficiency challenge by expanding the information collected per query to contain both preferences and optional text prompting. To accomplish this, we leverage the zero-shot capabilities of a large language model (LLM) to reason from the text provided by humans. To accommodate the additional query information, we reformulate the reward learning objectives to contain flexible highlights — state-action pairs that contain relatively high information and are related to the features processed in a zero-shot fashion from a pretrained LLM. In both a simulated scenario and a user study, we reveal the effectiveness of our work by analyzing the feedback and its implications. Additionally, the collective feedback collected serves to train a robot on socially compliant trajectories in a simulated social navigation landscape. We provide video examples of the trained policies at

Power in Human-Robot Interaction

  • Yoyo Tsung-Yu Hou
  • EunJeong Cheon
  • Malte F. Jung

Power is a fundamental determinant of social life, yet it remains elusive in Human-Robot Interaction (HRI). This paper unveils power’s pervasive but largely unexplored role in HRI by systematically investigating its varied manifestations across HRI literature. We first introduce definitions of power and then delve into the existing HRI literature through a lens of power, examining studies that directly address power and those exploring power-related social configurations and concepts such as authority, dominance, and status. Leveraging Fiske and Berdahl’s model and French and Raven’s bases of power framework, we also explore the nuances of power in many HRI studies where power is not explicitly addressed. Finally, we propose power as a core concept to advance HRI— explaining fragmented existing findings through a coherent theory and delineating a cohesive theoretical trajectory for future investigations.

“Give it Time:” Longitudinal Panels Scaffold Older Adults’ Learning and Robot Co-Design

  • Long-Jing Hsu
  • Philip B. Stafford
  • Weslie Khoo
  • Manasi Swaminathan
  • Kyrie Jig Amon
  • Hiroki Sato
  • Katherine M. Tsui
  • David J. Crandall
  • Selma Sabanović

Participatory robot design projects with older adults often use multiple sessions to encourage design feedback and active participation from users. Prior projects have, however, not analyzed the learning outcomes for older adults across co-design sessions and how they support constructive design feedback and meaningful participation. To bridge this gap, we examined the learning outcomes within a “longitudinal panel.” This panel comprised seven co-design sessions with 11 older adults of varying cognitive abilities over six months, aimed at designing a robot to guide a photograph-based conversational activity. Using Nelson and Stolterman’s framework of the hierarchy of design-learning, we demonstrate how older adult panelists achieved multiple design-learning outcomes- capacity, confidence, capability, competence, courage, and connection- which allowed them to provide actionable design suggestions. We provide guidelines for conducting longitudinal panels that can enhance user design-learning and participation in robot design.

(Gestures Vaguely): The Effects of Robots’ Use of Abstract Pointing Gestures in Large-Scale Environments

  • Annie Huang
  • Alyson Ranucci
  • Adam Stogsdill
  • Grace Clark
  • Keenan Schott
  • Mark Higger
  • Zhao Han
  • Tom Williams

As robots are deployed into large-scale human environments, they will need to engage in task-oriented dialogues about objects and locations beyond those that can currently be seen. In these contexts, speakers use a wide range of referring gestures beyond those used in the small-scale interaction contexts that HRI research typically investigates. In this work, we thus seek to understand how robots can better generate gestures to accompany their referring language in large-scale interaction contexts. In service of this goal, we present the results of two human-subject studies: (1) a human-human study exploring how human gestures change in large-scale interaction contexts, and to identify human-like gestures suitable to such contexts yet readily implemented on robot hardware; and (2) a human-robot study conducted in a tightly controlled Virtual Reality environment, to evaluate robots’ use of those identified gestures. Our results show that robot use of Precise Deictic and Abstract Pointing gestures afford different types of benefits when used to refer to visible vs. non-visible referents, leading us to formulate three concrete design guidelines. These results highlight both the opportunities for robot use of more humanlike gestures in large-scale interaction contexts, as well as the need for future work exploring their use as part of multi-modal communication.

Modeling Variation in Human Feedback with User Inputs: An Exploratory Methodology

  • Jindan Huang
  • Reuben M. Aronson
  • Elaine Schaertl Short

To expedite the development process of interactive reinforcement learning (IntRL) algorithms, prior work often uses perfect oracles as simulated human teachers to furnish feedback signals. These oracles typically derive from ground-truth knowledge or optimal policies, providing dense and error-free feedback to a robot learner without delay. However, this machine-like feedback behavior fails to accurately represent the diverse patterns observed in human feedback, which may lead to unstable or unexpected algorithm performance in real-world human-robot interaction. To alleviate this limitation of oracles in oversimplifying user behavior, we propose a method for modeling variation in human feedback that can be applied to a standard oracle. We present a model with 5 dimensions of feedback variation identified in prior work. This model enables the modification of feedback outputs from perfect oracles to introduce more human-like features. We demonstrate how each model attribute can impact on the learning performance of an IntRL algorithm through a simulation experiment. We also conduct a proof-of-concept study to illustrate how our model can be populated from people in two ways. The modeling results intuitively present the feedback variation among participants and help to explain the mismatch between oracles and human teachers. Overall, our method is a promising step towards refining simulated oracles by incorporating insights from real users.

Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control

  • Rajat Kumar Jenamani
  • Daniel Stabile
  • Ziang Liu
  • Abrar Anwar
  • Katherine Dimitropoulou
  • Tapomayukh Bhattacharjee

Robot-assisted feeding can greatly enhance the lives of those with mobility limitations. Modern feeding systems can pick up and position food in front of a care recipient’s mouth for a bite. However, many with severe mobility constraints cannot lean forward and need direct inside-mouth food placement. This demands precision, especially for those with restricted mouth openings, and appropriately reacting to various physical interactions – incidental contacts as the utensil moves inside, impulsive contacts due to sudden muscle spasms, deliberate tongue maneuvers by the person being fed to guide the utensil, and intentional bites. In this paper, we propose an inside-mouth bite transfer system that addresses these challenges with two key components: a multi-view mouth perception pipeline robust to tool occlusion, and a control mechanism that employs multimodal time-series classification to discern and react to different physical interactions. We demonstrate the efficacy of these individual components through two ablation studies. In a full system evaluation, our system successfully fed 13 care recipients with diverse mobility challenges. Participants consistently emphasized the comfort and safety of our inside-mouth bite transfer system, and gave it high technology acceptance ratings – underscoring its transformative potential in real-world scenarios. Supplementary materials and videos can be found at: \href .

Social Cue Detection and Analysis Using Transfer Entropy

  • Haoyang Jiang
  • Elizabeth A. Croft
  • Michael G. Burke

Robots that work close to humans need to understand and use social cues to act in a socially acceptable manner. Social cues are a form of communication (i.e., information flow) between people. In this paper, a framework is introduced to detect and analyse a class of perceptible social cues that are nonverbal and episodic, and the related information transfer using an information-theoretic measure, namely, transfer entropy. We use a group-joining setting to demonstrate the practicality of transfer entropy for analysing communications between humans. Then we demonstrate the framework in two settings involving social interactions between humans: object-handover and person-following. Our results show that transfer entropy can identify information flows between agents and when and where they occur. Potential applications of the framework include information flow or social cue analysis for interactive robot design and socially-aware robot planning.

A Comprehensive User Study on Augmented Reality-Based Data Collection Interfaces for Robot Learning

  • Xinkai Jiang
  • Paul Mattes
  • Xiaogang Jia
  • Nicolas Schreiber
  • Gerhard Neumann
  • Rudolf Lioutikov

Future versatile robots need the ability to learn new tasks and behaviors from demonstrations. Recent advances in virtual and augmented reality position these technologies as great candidates for the efficient and intuitive collection of large sets of demonstrations. While there are different possible approaches to control a virtual robot there has not yet been an evaluation of these control interfaces in regards to their efficiency and intuitiveness. These characteristics become particularly important when working with non-expert users and complex manipulation tasks. To this end, this work investigates five different interfaces to control a virtual robot in a comprehensive user study across various virtualized tasks in an AR setting. These interfaces include Hand Tracking, Virtual Kinesthetic Teaching, Gamepad and Motion Controller. Additionally, this work introduces Kinesthetic Teaching as a novel interface to control virtual robots in AR settings, where the virtual robot mimics the movement of a real robot manipulated by the user. This study reveals valuable insights into their usability and effectiveness. It shows that the proposed Kinesthetic Teaching interface significantly outperforms other interfaces in both objective and subjective metrics based on success rate, task completeness, and completion time and User Experience Questionnaires (UEQ+).

Constructing a Social Life with Robots: Shifting Away From Design Patterns Towards Interaction Ritual Chains

  • Waki Kamino
  • Malte F. Jung
  • Selma Sabanović

Robot designers commonly conceptualize robot sociality as a collection of features and capabilities. In contrast, sociologists define sociality as continuously constructed through interpersonal interactions. Based on the latter perspective, we trace how robots are incorporated into emerging social interaction ritual chains by robot companies and their staff and by robot owners across diverse contexts: homes, cafes, robot stores, user-organized meetups, and company events for robot users. Our empirical findings from ethnographic field work in Japan relating to three robots — aibo, RoboHon, and LOVOT — show how companies create positive interactions between people and robots by incorporating familiar design patterns into robots, modeling successful interactions in person and online, and bringing owners together in events that establish common values of acceptance of social robots as artifacts to live with and nurture. Owners, for their part, develop interaction rituals that include robots in their daily activities, make interpersonal connections, and experience emotionally resonant interactions, around robots in public meetups and events. Through these emerging interaction ritual chains, companies and owners construct the notion of robots as social agents to live with as a meaningful component of their emotional experiences and broader social relationships. Our work suggests that social robot design should consider this broader framing of sociality and create affordances for establishing interaction ritual chains more explicitly.

Collabot: A Robotic System That Assists Library Users Through Collaboration Between Robots

  • Dahyun Kang
  • Haeri Hwang
  • Sonya S. Kwak

A library serves as a repository of knowledge accessible to individuals of all ages, genders, educational backgrounds, social statuses, and economic levels. It stands as a communal space where community members can gather, bridging information disparities among various societal strata. To enhance accessibility to such libraries for a broader spectrum of people, we have introduced the CollaBot system. This system offers tailored services to users through the collaboration of robots. Our investigation encompassed the acceptance of robot types by users, robot characterization, and the prioritization of robot-provided services. Over the course of three stages of user evaluation, it became evident that participants preferred product-type robots over anthropomorphic robots. Furthermore, they expressed a preference for robots that assist other robots, even if these assisting robots exhibit clumsiness, as opposed to robots that exclusively excel in their designated tasks. Lastly, service prioritization varied based on the specific limitations or deficiencies faced by individual users.

Alchemist: LLM-Aided End-User Development of Robot Applications

  • Ulas Berk Karli
  • Juo-Tung Chen
  • Victor Nikhil Antony
  • Chien-Ming Huang

Large Language Models (LLMs) have the potential to catalyze a paradigm shift in end-user robot programming—moving from the conventional process of user specifying programming logic to an iterative, collaborative process in which the user specifies desired program outcomes while LLM produces detailed specifications. We introduce a novel integrated development system, Alchemist, that leverages LLMs to empower end-users in creating, testing, and running robot programs using natural language inputs, aiming to reduce the required knowledge for developing robot applications. We present a detailed examination of our system design and provide an exploratory study involving true end-users to assess capabilities, usability, and limitations of our system. Through the design, development, and evaluation of our system, we derive a set of lessons learned from the use of LLMs in robot programming. We discuss how LLMs may be the next frontier for democratizing end-user development of robot applications.

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

  • Callie Y. Kim
  • Christine P. Lee
  • Bilge Mutlu

Large-language models (LLMs) hold significant promise in improving human-robot interaction, offering advanced conversational skills and versatility in managing diverse, open-ended user requests in various tasks and domains. Despite the potential to transform human-robot interaction, very little is known about the distinctive design requirements for utilizing LLMs in robots, which may differ from text and voice interaction and vary by task and context. To better understand these requirements, we conducted a user study (n = 32) comparing an LLM-powered social robot against text- and voice-based agents, analyzing task-based requirements in conversational tasks, including choose, generate, execute, and negotiate. Our findings show that LLM-powered robots elevate expectations for sophisticated non-verbal cues and excel in connection-building and deliberation, but fall short in logical communication and may induce anxiety. We provide design implications both for robots integrating LLMs and for fine-tuning LLMs for use with robots.

A Taxonomy of Robot Autonomy for Human-Robot Interaction

  • Stephanie Kim
  • Jacy Reese Anthis
  • Sarah Sebo

Robot autonomy is an influential and ubiquitous factor in human-robot interaction (HRI), but it is rarely discussed beyond a one-dimensional measure of the degree to which a robot operates without human intervention. As robots become more sophisticated, this simple view of autonomy could be expanded to capture the variety of autonomous behaviors robots can exhibit and to match the rich literature on human autonomy in philosophy, psychology, and other fields. In this paper, we conduct a systematic literature review of robot autonomy in HRI and integrate this with the broader literature into a taxonomy of six distinct forms of autonomy: those based on robot and human involvement at runtime (operational autonomy, intentional autonomy, shared autonomy), human involvement before runtime (non-deterministic autonomy), and expressions of autonomy at runtime (cognitive autonomy, physical autonomy). We discuss future considerations for autonomy in HRI that emerge from this study, including moral consequences, the idealization of “full” robot autonomy, and connections to agency and free will.

Iterative Robot Waiter Algorithm Design: Service Expectations and Social Factors

  • Heather Knight
  • Deanna Flynn
  • Theing Mwe Oo
  • Julia Hansen

Mobile robots carrying food in restaurants are here. What service behavior norms do people expect them to follow? This paper evaluates robot waiter algorithms and service parameters for a robot serving two participants at a simulated cocktail event, varying body-storming inspired context variables such as: “hunger level” and “relationship to each other,” robot delivery algorithms (lead, follow, ambient), and participant pose (standing, seated). In the within-subjects design, pairs of people were given a series of context prompts, and told to participate as felt natural. Output variables included whether they took food and post-trial survey ratings of the robot. The results show a positive correlation between food taking (or feelings of obligation to take food) and human OR robot initiative, relative to a mixed-ambient algorithm with no explicit leader. The robot waiter that initiates is the clearest and most noticeable. There were also some challenges: people in conversation would sometimes forget or delay calls for cupcakes, ambient robot motion was hardest to notice, and bringing food one person ordered to the other was unforgivable. When in doubt, go to the middle. Finally, participants enjoyed the robot spinning, describing it as a dessert tray which attracted their eyes to the robot.

Sprout: Designing Expressivity for Robots Using Fiber-Embedded Actuator

  • Amy Koike
  • Michael Wehner
  • Bilge Mutlu

In this paper, we explore how techniques from soft robotics can help create a new form of robot expression. We present Sprout, a soft expressive robot that conveys its internal states by changing its body shape. Sprout can extend, bend, twist, and expand using fiber-embedded actuators integrated into its construction. These deformations enable Sprout to express its internal states, for example, by expanding to express anger and bending its body sideways to express curiosity. Through two user studies, we investigated how users interpreted Sprout’s expressions, their perceptions of Sprout, and their expectations from future iterations of Sprout’s design. We argue that the use of soft actuators opens a novel design space for robot expressions to convey internal states, emotions, and intent.

Reactive or Proactive? How Robots Should Explain Failures

  • Gregory LeMasurier
  • Alvika Gautam
  • Zhao Han
  • Jacob W. Crandall
  • Holly A. Yanco

As robots tackle increasingly complex tasks, the need for explanations becomes essential for gaining trust and acceptance. Explainable robotic systems should not only elucidate failures when they occur but also predict and preemptively explain potential issues. This paper compares explanations from Reactive Systems, which detect and explain failures after they occur, to Proactive Systems, which predict and explain issues in advance. Our study reveals that the Proactive System fosters higher perceived intelligence and trust and its explanations were rated more understandable and timely. Our findings aim to advance the design of effective robot explanation systems, allowing people to diagnose and provide assistance for problems that may prevent a robot from finishing its task.

Snitches Get Unplugged: Adolescents’ Privacy Concerns about Robots in the Home are Relationally Situated

  • Leigh Levinson
  • Christena Nippert-Eng
  • Randy Gomez
  • Selma Sabanović

Though teens are a population with growing agency and use of smart technologies, their concerns surrounding privacy with AI and robots are under-represented in research. Using focus group discussions and a mixed methods analysis, we present teens’ comfort levels with robotic information collection and sharing during three hypothetical scenarios involving a child interacting with the Haru social robot in the home. We find participant concerns align with an access-based definition of privacy which prioritizes being in control of their information and of when the robot behaves autonomously. Responses also indicate that teens conceptualize Haru not just as an intelligent device, but also as a social entity. Their shifts in comfort and discussions reflect an engagement in social relationship management with robots in the home in cases where the robot mediates a user’s responsibilities and relationships with others.

Back to School – Sustaining Recurring Child-Robot Educational Interactions After a Long Break

  • Mike E.U. Ligthart
  • Simone M. De Droog
  • Marianne Bossema
  • Lamia Elloumi
  • Mirjam de Haas
  • Matthijs H.J. Smakman
  • Koen V. Hindriks
  • Somaya Ben Allouch

Maintaining the child-robot relationship after a significant break, such as a holiday, is an important step for developing sustainable social robots for education. We ran a four-session user study (n = 113 children) that included a nine-month break between the third and fourth session. During the study, participants practiced math with the help of a social robot math tutor. We found that social personalization is an effective strategy to better sustain the child-robot relationship than the absence of social personalization. To become reacquainted after the long break, the robot summarizes a few pieces of information it had stored about the child. This gives children a feeling of being remembered, which is a key contributor to the effectiveness of social personalization. Enabling the robot to refer to information previously shared by the child is another key contributor to social personalization. Conditional for its effectiveness, however, is that children notice these memory references. Finally, although we found that children’s interest in the tutoring content is related to relationship formation, personalizing the topics did not lead to more interest in the content. It seems likely that not all of the memory information that was used to personalize the content was up-to-date or socially relevant.

Improving Explainable Object-induced Model through Uncertainty for Automated Vehicles

  • Shihong Ling
  • Yue Wan
  • Xiaowei Jia
  • Na Du

The rapid evolution of automated vehicles (AVs) has the potential to provide safer, more efficient, and comfortable travel options. However, these systems face challenges regarding reliability in complex driving scenarios. Recent explainable AV architectures neglect crucial information related to inherent uncertainties while providing explanations for actions. To overcome such challenges, our study builds upon the “object-induced” model approach that prioritizes the role of objects in scenes for decision-making and integrates uncertainty assessment into the decision-making process using an evidential deep learning paradigm with a Beta prior. Additionally, we explore several advanced training strategies guided by uncertainty, including uncertainty-guided data reweighting and augmentation. Leveraging the BDD-OIA dataset, our findings underscore that the model, through these enhancements, not only offers a clearer comprehension of AV decisions and their underlying reasoning but also surpasses existing baselines across a broad range of scenarios.

Modelling Experts’ Sampling Strategy to Balance Multiple Objectives During Scientific Explorations

  • Shipeng Liu
  • Cristina G. Wilson
  • Zachary I. Lee
  • Feifei Qian

Our analysis of human sampling decision data reveals that scientists adapt their sampling strategies to balance multiple objectives based on two key factors: the current level of information about the environment, and the availability of sampling location options with large potential rewards. While this work is only a beginning step towards the development of cognitive-compatible robotic decision algorithms, our findings show by better understanding human decision processes, robots can use extremely simple algorithms to connect experts’ high-level objectives to desired sampling locations while balancing multiple objectives. Going forward, exploring how humans coordinate and prioritize multiple objectives under more sophisticated scientific exploration scenarios, such as with multiple competing hypotheses, with hypotheses regarding multiple variables, or with additional sampling objectives, would be helpful to explore. These understandings could help our robots produce explainable sampling strategies that are well-aligned with humans’ high level goals, and improve humans’ trust and confidence during teaming. These cognitive understandings could also allow robots to identify potential vulnerabilities in human decisions, such as biases and fatigue, and provide targeted support to enhance scientific outcomes. In addition, we expect that these cognitive insights could complement existing robotic decision methods by informing which algorithms to use, and eventually empower robots to become intelligent teammates that can truly participate in the decision-making process.

RoboVisAR: Immersive Authoring of Condition-based AR Robot Visualisations

  • Rasmus Skovhus Lunding
  • Mille Skovhus Lunding
  • Tiare Feuchtner
  • Marianne Graves Petersen
  • Kaj Grønbæk
  • Ryo Suzuki

We introduce RoboVisAR, an immersive augmented reality (AR) authoring tool for in-situ robot visualisations. AR robot visualisations, such as the robot’s movement path, status, and safety zones, have been shown to benefit human-robot collaboration. However, their creation requires extensive skills in both robotics and AR programming. To address this, RoboVisAR allows users to create custom AR robot visualisations without programming. By recording an example robot behaviour, users can design, combine, and test visualisations in-situ within a mixed reality environment. RoboVisAR currently supports six types of visualisations (Path, Point of Interest, Safety Zone, Robot State, Message, Force/Torque) and four types of conditions for when they are displayed (Robot State, Proximity, Box, Force/Torque). With this tool, users can easily present different visualisations on demand and make them context-aware to avoid visual clutter. An expert user study with three participants suggests that users appreciate the customizability of the visualisations, which could easily be authored in less than ten minutes.

RABBIT: A Robot-Assisted Bed Bathing System with Multimodal Perception and Integrated Compliance

  • Rishabh Madan
  • Skyler Valdez
  • David Kim
  • Sujie Fang
  • Luoyan Zhong
  • Diego T. Virtue
  • Tapomayukh Bhattacharjee

This paper introduces RABBIT, a novel robot-assisted bed bathing system designed to address the growing need for assistive technologies in personal hygiene tasks. It combines multimodal perception and dual (software and hardware) compliance to perform safe and comfortable physical human-robot interaction. Using RGB and thermal imaging to segment dry, soapy, and wet skin regions accurately, RABBIT can effectively execute washing, rinsing, and drying tasks in line with expert caregiving practices. Our system includes custom-designed motion primitives inspired by human caregiving techniques, and a novel compliant end-effector called Scrubby, optimized for gentle and effective interactions. We conducted a user study with 12 participants, including one participant with severe mobility limitations, demonstrating the system’s effectiveness and perceived comfort. Supplementary material and videos can be found on our website \href .

Generative Expressive Robot Behaviors using Large Language Models

  • Karthik Mahadevan
  • Jonathan Chien
  • Noah Brown
  • Zhuo Xu
  • Carolina Parada
  • Fei Xia
  • Andy Zeng
  • Leila Takayama
  • Dorsa Sadigh

People employ expressive behaviors to effectively communicate and coordinate their actions with others, such as nodding to acknowledge a person glancing at them or saying “excuse me” to pass people in a busy corridor. We would like robots to also demonstrate expressive behaviors in human-robot interaction. Prior work proposes rule-based methods that struggle to scale to new communication modalities or social situations, while data-driven methods require specialized datasets for each social situation the robot is used in. We propose to leverage the rich social context available from large language models (LLMs) and their ability to generate motion based on instructions or user preferences, to generate expressive robot motion that is adaptable and composable, building upon each other. Our approach utilizes few-shot chain-of-thought prompting to translate human language instructions into parametrized control code using the robot’s available and learned skills. Through user studies and simulation experiments, we demonstrate that our approach produces behaviors that users found to be competent and easy to understand. Supplementary material can be found at

Presentation of Robot-Intended Handover Position using Vibrotactile Interface during Robot-to-Human Handover Task

  • Muhammad Akmal Bin Mohammed Zaffir
  • Takahiro Wada

Advancements in robot autonomy and safety have enabled close interactions, such as object handovers, with humans. During robot-to-human handovers in assembly tasks, the robot considers the state of the human to determine its optimal handover position and timing. However, humans may struggle to focus on their primary tasks because of the need to track the robot’s movement. This study aims to develop a vibrotactile interface that helps humans maintain focus on their primary tasks during object reception. The interface conveys the robot-intended handover position on the human forearm by displaying the angular direction and distance relative to the human hand via vibrotactile cues. The experimental results demonstrated that the interface allowed participants to receive objects with faster reactions and completion times, with reduced head rotation towards the robot. Participants also subjectively perceived improved performance and reduced mental workload compared with the condition without the interface.

What a Thing to Say! Which Linguistic Politeness Strategies Should Robots Use in Noncompliance Interactions?

  • Terran Mott
  • Aaron Fanganello
  • Tom Williams

For social robots to succeed in human environments, they must respond in effective yet appropriate ways when humans violate social and moral norms, e.g., when humans give them unethical commands. Humans expect robots to be competent and proportional in their norm violation responses, and there are a wide range of strategies robots could use to tune the politeness of their utterances to achieve effective, yet appropriate responses. Yet it is not obvious whether all such strategies are suitable for robots to use. In this work, we assess a robot’s use of human-like Face Theoretic linguistic politeness strategies. Our results show that while people expect robots to modulate the politeness of their responses, they do not expect them to strictly mimic human linguistic behaviors. Specifically, linguistic politeness strategies that use direct, formal language are perceived as more effective and more appropriate than strategies that use indirect, informal language.

“I’m Not Touching You. It’s The Robot!”: Inclusion Through A Touch-Based Robot Among Mixed-Visual Ability Children

  • Isabel Neto
  • Yuhan Hu
  • Filipa Correia
  • Filipa Rocha
  • João Nogueira
  • Katharina Buckmayer
  • Guy Hoffman
  • Hugo Nicolau
  • Ana Paiva

Children with visual impairments often struggle to fully participate in group activities due to limited access to visual cues. They have difficulty perceiving what is happening, when, and how to act-leading to children with and without visual impairments being frustrated with the group activity, reducing mutual interactions. To address this, we created Touchibo, a tactile storyteller robot acting in a multisensory setting, encouraging touch-based interactions. Touchibo provides an inclusive space for group interaction as touch is a highly accessible modality in a mixed-visual ability context. In a study involving 107 children (37 with visual impairments), we compared Touchibo to an audio-only storyteller. Results indicate that Touchibo significantly improved children’s individual and group participation perception, sparking touch-based interactions and the storyteller was more likable and helpful. Our study highlights touch-based robots’ potential to enrich children’s social interactions by prompting interpersonal touch, particularly in mixed-visual ability settings.

Role-Playing with Robot Characters: Increasing User Engagement through Narrative and Gameplay Agency

  • Spencer Ng
  • Ting-Han Lin
  • You Li
  • Sarah Sebo

Live entertainment is moving towards a greater participatory culture, with dynamic narratives told through audience interaction. Robot characters offer a unique opportunity to mitigate the challenges of creating personalized entertainment at scale. However, robots often cannot react to audience responses, limiting opportunities for audience participation. In this work, we explore methods to increase user agency in live entertainment experiences with robot characters to improve user engagement and enjoyment. In a between-subjects study (N=60), we create an immersive story where users role-play as detectives with two distinct robot characters. Users either (1) have greater involvement and self-identification in the story by talking with the robots in-character (narrative condition), (2) have a more active role in solving puzzles (gameplay condition), or (3) follow along without being prompted by the robots for input (control condition). Our results show that increasing user agency in a role-playing experience, in either its narrative or its gameplay, improves users’ flow state, sense of autonomy and competence, verbal engagement, and perceptions of the robot characters’ engagement. Increasing narrative agency also led to longer unprompted reactions from participants, while gameplay agency improved feelings of immersion and relatedness with the robots. These findings suggest that creating either narrative or gameplay agency can improve user engagement, which can extend to broader robot interactions where gameplay elements and role-playing in stories can be incorporated.

Design and Evaluation of a Socially Assistive Robot Schoolwork Companion for College Students with ADHD

  • Amy O’Connell
  • Ashveen Banga
  • Jennifer Ayissi
  • Nikki Yaminrafie
  • Ellen Ko
  • Andrew Le
  • Bailey Cislowski
  • Maja Mataric

College students with ADHD respond positively to simple socially assistive robots (SARs) that monitor attention and provide non-verbal feedback, but studies have been done only in brief in-lab sessions. We present an initial design and evaluation of an in-dorm SAR study companion for college students with ADHD. This work represents the introductory stages of an ongoing user-centered, participatory design process. In a three-week within-subjects user study, university students (N=11) with self-reported symptoms of adult ADHD had a SAR study companion in their dorm room for two weeks and a computer-based system for one week. Toward developing SARs for long-term, in-dorm use, we focus on 1) evaluating the usability and desire for SAR study companions by college students with ADHD, and 2) collecting participant feedback about the SAR design and functionality. Participants responded positively to the robot; after one week of regular use, 91% (10 of 11) chose to continue using the robot voluntarily in the second week.

Independence in the Home: A Wearable Interface for a Person with Quadriplegia to Teleoperate a Mobile Manipulator

  • Akhil Padmanabha
  • Janavi Gupta
  • Chen Chen
  • Jehan Yang
  • Vy Nguyen
  • Douglas J Weber
  • Carmel Majidi
  • Zackory Erickson

Teleoperation of mobile manipulators within a home environment can significantly enhance the independence of individuals with severe motor impairments, allowing them to regain the ability to perform self-care and household tasks. There is a critical need for novel teleoperation interfaces to offer effective alternatives for individuals with impairments who may encounter challenges in using existing interfaces due to physical limitations. In this work, we iterate on one such interface, HAT (Head-Worn Assistive Teleoperation), an inertial-based wearable integrated into any head-worn garment. We evaluate HAT through a 7-day in-home study with Henry Evans, a non-speaking individual with quadriplegia who has participated extensively in assistive robotics studies. We additionally evaluate HAT with a proposed shared control method for mobile manipulators termed Driver Assistance and demonstrate how the interface generalizes to other physical devices and contexts. Our results show that HAT is a strong teleoperation interface across key metrics including efficiency, errors, learning curve, and workload. Code and videos are located on our project website.

What Is Your Other Hand Doing, Robot? A Model of Behavior for Shopkeeper Robot’s Idle Hand

  • Xiang Pan
  • Malcolm Doering
  • Takayuki Kanda

In retail settings, a robot’s one-handed manipulation of objects can come across as thoughtless and impolite, thus creating a negative customer experience. To solve this problem, we first observed how human shopkeepers interact with customers, specifically focusing on their hand movements during object manipulation. From the observation and analysis of shopkeepers’ hand movements, we identified an essential element of their idle hand movements: “support” provided by the idle hand as the primary hand manipulates an object. Based on this observation, we proposed a model that coordinates the movements of a robot’s idle hand with its primary task-engaged hand, emphasizing its supportive behaviors. In a within-subjects study, 20 participants interacted with robot shopkeepers under different conditions to assess the impact of incorporating support behavior with the idle hand. The results show that the proposed model significantly outperforms a baseline in terms of politeness and competence, suggesting enhanced object-based interactions between the robot shopkeepers and customers.

Encountering Autonomous Robots on Public Streets

  • Hannah R. M. Pelikan
  • Stuart Reeves
  • Marina N. Cantarutti

Robots deployed in public settings enter spaces that humans live and work in. Studies of HRI in public tend to prioritise direct and deliberate interactions. Yet this misses the most common form of response to robots, which ranges from subtle fleeting interactions to virtually ignoring them. Taking an ethnomethodological approach building on video recordings, we show how robots become embedded in urban spaces both from a perspective of the social assembly of the physical environment (the streetscape) and the socially organised nature of everyday street life. We show how such robots are effectively ‘granted passage’ through these spaces as a result of the practical work of the streets’ human inhabitants. We detail the contingent nature of the streetscape, drawing attention to its various members and the accommodation work they are doing. We demonstrate the importance of studying robots during their whole deployment, and approaches that focus on members’ interactional work.

Preference-Conditioned Language-Guided Abstraction

  • Andi Peng
  • Andreea Bobu
  • Belinda Z. Li
  • Theodore R. Sumers
  • Ilia Sucholutsky
  • Nishanth Kumar
  • Thomas L. Griffiths
  • Julie A. Shah

Learning from demonstrations is a common way for users to teach robots, but it is prone to spurious feature correlations. Recent work constructs state abstractions, i.e. visual representations containing task-relevant features, from language as a way to perform more generalizable learning. However, these abstractions also depend on a user’s preference for what matters in a task, which may be hard to describe or infeasible to exhaustively specify using language alone. How do we construct abstractions to capture these latent preferences? We observe that how humans behave reveals how they see the world. Our key insight is that changes in human behavior inform us that there are differences in preferences for how humans see the world, i.e. their state abstractions. In this work, we propose using language models (LMs) to query for those preferences directly given knowledge that a change in behavior has occurred. In our framework, we use the LM in two ways: first, given a text description of the task and knowledge of behavioral change between states, we query the LM for possible hidden preferences; second, given the most likely preference, we query the LM to construct the state abstraction. In this framework, the LM is also able to ask the human directly when uncertain about its own estimate. We demonstrate our framework’s ability to construct effective preference-conditioned abstractions in simulated experiments, a user study, as well as on a real Spot robot performing mobile manipulation tasks.

Goal-Oriented End-User Programming of Robots

  • David Porfirio
  • Mark Roberts
  • Laura M. Hiatt

End-user programming (EUP) tools must balance user control with the robot’s ability to plan and act autonomously. Many existing task-oriented EUP tools enforce a specific level of control, e.g., by requiring that users hand-craft detailed sequences of actions, rather than offering users the flexibility to choose the level of task detail they wish to express. We thereby created a novel EUP system, Polaris, that in contrast to most existing EUP tools, uses goal predicates as the fundamental building block of programs. Users can thereby express high-level robot objectives or lower-level checkpoints at their choosing, while an off-the-shelf task planner fills in any remaining program detail. To ensure that goal-specified programs adhere to user expectations of robot behavior, Polaris is equipped with a Plan Visualizer that exposes the planner’s output to the user before runtime. In what follows, we describe our design of Polaris and its evaluation with 32 human participants. Our results support the Plan Visualizer’s ability to help users craft higher-quality programs. Furthermore, there are strong associations between user perception of the robot and Plan Visualizer usage, and evidence that robot familiarity has a key role in shaping user experience.

Zero-Shot Learning to Enable Error Awareness in Data-Driven HRI

  • Joshua Ravishankar
  • Malcolm Doering
  • Takayuki Kanda

Data-driven social imitation learning is a minimally-supervised approach to generating robot behaviors for human-robot interaction (HRI). However, this type of learning-based approach is error-prone. Existing error detection methods for HRI rely on data labeling, rendering them inappropriate for the data-driven paradigm. We present a zero-shot error detection strategy that requires no labeled data. We use human interaction data to learn models of normal human behavior, then use these models to extract features that help discriminate abnormal human reactions to robot errors. In this feature space, we frame error detection as a novelty detection task, utilizing human interaction data to learn a model of non-erroneous interactions in an unsupervised fashion. Then, we apply the fitted novelty detector to HRI data to identify erroneous robot behavior. We show that our method obtains an average precision of 0.497 on errors, outperforming unsupervised baselines and supervised approaches with limited training data.

Lie, Repent, Repeat: Exploring Apologies after Repeated Robot Deception

  • Kantwon Rogers
  • Reiden John Allen Webber
  • Jinhee Chang
  • Geronimo Gorostiaga Zubizarreta
  • Ayanna Howard

This work presents an empirical study of repeated robot deception and its effects on changes in behavior and trust in a human-robot interaction scenario. 715 online and 50 in-person participants completed a multitrial driving simulation in which the car’s robot assistant repeatedly lies and apologizes. Through a mixed-method approach, our results show that apologies that offer justifications for deception in our scenario mitigate the negative effects on trust over multiple trials. However, given the time-sensitive, high-risk nature of our scenario, none of the apologies caused people to significantly change their decision to exceed the speed limit while rushing their dying friend to the hospital. These results add much needed knowledge to the understudied area of robot deception and could inform designers and policymakers of future practices when considering deploying robots that may learn to deceive.

With Every Breath: Testing the Effects of Soft Robotic Surfaces on Attention and Stress

  • Elena Sabinson
  • Jack Neiberg
  • Keith Evan Green

We report on the effects of a novel soft robot of our design on emotional wellbeing. Participants (N=94) engaged with our soft robotic surface designed to simulate the benefits of nature and provide a therapeutic behavioral intervention. The study assessed group differences in attention, perceived restorativeness, and self-reported stress between three groups: a group that performed a breathing exercise with the robot, a group that watched the robot perform an ocean-inspired movement designed to capture involuntary attention, and a control where the robot was static. The Breathing Group had a significant reduction in self-reported stress compared to the Control Group. Significant differences in attention and perceived restoration were not found. Qualitative feedback suggested the robot did provide a positive distraction in the environment and participants were generally favorable to the robot, characterizing it as soothing and fascinating. Feedback on the sensory qualities showed that people who did not initially enjoy the texture or sound often acclimated to the novelty of the surface with improved perceptions over time. These findings suggest the promise of soft robots to support mental wellbeing.

Are You Sure? – Multi-Modal Human Decision Uncertainty Detection in Human-Robot Interaction

  • Lisa Scherf
  • Lisa Alina Gasche
  • Eya Chemangui
  • Dorothea Koert

In a question-and-answer setting, the respondent is often not only communicating the requested information but also indicating their confidence in the answer through various behavioral cues. Humans excel at interpreting these cues and monitoring the uncertainty of other persons. Being able to detect human uncertainty in human-robot interactions in a similar way can enable future robotic systems to better recognize uncertain and error-prone human input. Additionally, automatic human uncertainty detection can enhance the responsiveness of robots to the user in moments of uncertainty by providing help or clarification. While there is some work on uncertainty detection based on a single modality, only a few works focus on multi-modal uncertainty detection. Even fewer works explore how human uncertainty manifests through behavioral cues in human-robot interactions. In this work, we analyze occurrences of behavioral cues related to self-reported uncertainty on experimental data from 27 participants across two decision-making tasks. Additionally, in the first task, we varied if participants interacted with a human or a robot. On the recorded data, we extract features accessible via a webcam and a microphone and train a multi-modal classifier. Experimental evaluation of our developed classifier shows that it significantly outperforms third-person annotators in accuracy and F1 score. Humans report feeling less observed when responding to a robot compared to a human. Nevertheless, we found that the behavioral differences did not significantly affect the performance of our proposed uncertainty classification.

Navigating Real-World Complexity: A Multi-Medium System for Heterogeneous Robot Teams and Multi-Stakeholder Human-Robot Interaction

  • Pete Schroepfer
  • Jan P. Gründling
  • Nathalie Schauffel
  • Simon Oehrl
  • Sebastian Pape
  • Torsten W. Kuhlen
  • Benjamin Weyers
  • Thomas Ellwart
  • Cédric Pradalier

Real-world robot system deployment is often performed in complex and unstructured environments. These complex environments coupled with multi-faceted global tasks often lead to complicated stakeholder structures, making designing for these environments extremely challenging. Magnifying this difficulty, tasks performed in these environments often cannot be accomplished by a single robot or even single robot type because of the broad range of needs and psychical constraints of the robots. In these cases, heterogeneous robot teams may need to be coupled to human team members to perform the global tasks. From a Human-Robot Interaction (HRI) perspective, this increases the complexity of designing and deploying the system significantly, as now complicated stakeholder structures are mixed with complex robot teams. This paper presents a novel real-world system and interface design leveraging multiple mediums to balance stakeholder needs. To this end, the UI presented here incorporates features that support shared mental models (SMMs), trust establishment and development, and utilizes a centralized data distribution architecture to improve team performance. In addition to the interface, this paper presents a detailed look at the design process and the lessons learned from the perspective of a multi-year, real-world deployed system, as part of a large European project consisting of 21 partners from varying countries and backgrounds.

Online Behavior Modification for Expressive User Control of RL-Trained Robots

  • Isaac Sheidlower
  • Mavis Murdock
  • Emma Bethel
  • Reuben M. Aronson
  • Elaine Schaertl Short

Reinforcement Learning (RL) is an effective method for robots to learn tasks. However, in typical RL, end-users have little to no control over how the robot does the task after the robot has been deployed. To address this, we introduce the idea of online behavior modification, a paradigm in which users have control over behavior features of a robot in real-time as it autonomously completes a task using an RL-trained policy. To show the value of this user-centered formulation for human-robot interaction, we present a behavior-diversity–based algorithm, Adjustable Control Of RL Dynamics (ACORD), and demonstrate its applicability to online behavior modification in simulation and a user study. In the study (n =23), users adjust the style of paintings as a robot traces a shape autonomously. We compare \algoshort to RL and Shared Autonomy (SA), and show \algoshort affords user-preferred levels of control and expression, comparable to SA, but with the potential for autonomous execution and robustness of RL. The code for this paper is available at

Artificial Emotions and the Evolving Moral Status of Social Robots

  • Arianna Sica
  • Henrik S. Sætra

This article aims to explore the potential impact of artificial emotional intelligence (AEI) on the ethical standing of social robots. By examining how AEI interacts with and potentially reshapes the two dominant perspectives on robots’ moral status, namely the property-oriented approach and the social-relational approach, we aim to offer fresh insights into this pressing dilemma. Our analysis reveals that although the incorporation of AEI does not conclusively confer moral status to current social robots, it might challenge the boundaries that separate robots from other entities customarily considered to have more status, thereby increasing the complexity of the debate.

Towards Balancing Preference and Performance through Adaptive Personalized Explainability

  • Andrew Silva
  • Pradyumna Tambwekar
  • Mariah Schrum
  • Matthew Gombolay

As robots and digital assistants are deployed in the real world, these agents must be able to communicate their decision-making criteria to build trust, improve human-robot teaming, and enable collaboration. While the field of explainable artificial intelligence (xAI) has made great strides to enable such communication, these advances often assume that one xAI approach is ideally suited to each problem (e.g., decision trees to explain how to triage patients in an emergency or feature-importance maps to explain radiology reports). This fails to recognize that users have diverse experiences or preferences for interaction modalities. In this work, we present two user-studies set in a simulated autonomous vehicle (AV) domain. We investigate (1) population-level preferences for xAI and (2) personalization strategies for providing robot explanations. We find significant differences between xAI modes (language explanations, feature-importance maps, and decision trees) in both preference (p < 0.01) and performance (p < 0.05). We also observe that a participant’s preferences do not always align with their performance, motivating our development of an adaptive personalization strategy to balance the two. We show that this strategy yields significant performance gains (p < 0.05), and we conclude with a discussion of our findings and implications for xAI in human-robot interactions.

Children’s Word Learning from Socially Contingent Robots Under Active vs. Passive Learning Conditions

  • Fatih Sivridag
  • Nivedita Mani

Language is learned through social interactions, in which gaze has a special role because it can be used to guide the attention and reference objects easily. Children, starting from very early ages, are also very good at utilizing gaze to map labels to referenced objects. To achieve language teaching robots, we need to understand how these functions of gaze can be implemented most efficiently. To this aim, we allowed children to interact with a social robot to learn the labels of several objects in a naturalistic setting. In some trials the child guided the gaze and chose the object to be learned while the robot was following and in the others they changed the roles and robot guided the gaze and decided on the object to be learned. We measured how much children actually followed the robot’s gaze and how many words they learned in these two conditions, referred to as active and passive learning conditions, respectively. The results indicate that although children followed the robot’s gaze and learned words successfully, there were no meaningful differences in word learning between the two conditions. The rate of gaze following and time spent looking at the robot did not influence word learning, either. The implications of these results for use of robots in educational settings are further discussed.

Driving a Ballbot Wheelchair with Hands-Free Torso Control

  • Seung Yun Song
  • Nadja Marin
  • Chenzhang Xiao
  • Mahshid Mansouri
  • Joao Ramos
  • Yu Chen
  • Adam W. Bleakney
  • Deana C. Mcdonagh
  • William R. Norris
  • Jeannette R. Elliott
  • Patricia B. Malik
  • Elizabeth T. Hsiao-Wecksler

A novel wheelchair called PURE (Personalized Unique Rolling Experience) that uses hands-free (HF) torso lean-to-steer control has been developed for manual wheelchair users (mWCUs). PURE addresses limitations of current wheelchairs, such as the inability to use both hands for life experiences instead of propulsion. PURE uses a ball-based robot drivetrain to offer a compact, self-balancing, omnidirectional mobile device. A custom sensor system converts rider torso motions into direction and speed commands to control PURE, which is especially useful if a rider has minimal torso range of motion. We explored whether PURE’s HF control performed as well as a traditional joystick (JS) human-robot interface and mWCUs, who may have reduced torso motion, performed as well as able-bodied users (ABUs). 10 mWCUs and 10 ABUs were trained and tested to drive PURE through courses replicating indoor environments. Each participant adjusted personal sensitivity settings for both HF and JS control. Repeated-measures MANOVA tests suggested that the effectiveness (number of collisions), efficiency (completion time), comfort (NASA TLX scores except physical demand), and robustness (index of performances) were similar for HF and JS control and between mWCUs and ABUs for all sections. These results suggest that PURE provides an effective method for controlling this new omnidirectional wheelchair by only using torso motion thus leaving both hands to be used for other tasks during propulsion.

Combining Emotional Gestures, Sound Effects, and Background Music for Robotic Storytelling – Effects on Storytelling Experience, Emotion Induction, and Robot Perception

  • Sophia C. Steinhaeusser
  • Ramona Piller
  • Birgit Lugrin

Storytelling is a long-established human tradition for entertainment and knowledge transfer. Social robots are emerging as a new storytelling medium, being able to imitate human storytelling using gestures but also extend it by adding, e.g., sound effects to the experience. Due to COVID-19 restrictions, we conducted an online video-based study to investigate the effects of congruent respectively incongruent or no gesture usage in combination with additional non-speech sounds, i.e. sound effects and background music, on recipients’ transportation into the story told, emotion induction, and perception of the robot. Results indicate no effect of additional non-speech sound integration on the variables listed above. Contradicting with related findings from in-person studies, we found a no significant differences between congruent, incongruent and no gesture usage. Last, no interplay of additional sounds and gesture congruence was identified. Future studies should provide deeper insights into the importance of multimodal congruence in video-taped robots and the possible advantages of adding non-speech sounds to online but also in-person robotic storytelling as well as their interplay in in-person HRI.

More Than Binary: Transgender and Non-binary Perspectives on Human Robot Interaction

  • Michael Stolp-Smith
  • Tom Williams

Research has shown that gendered robot designs prompt users to carry their gender biases into human-robot interactions. Yet avoiding gendered designs in human-robot interaction may be infeasible, as humans readily gender robots based on factors like name, voice, and pronouns. One solution to this challenge could be to use an intentionally agender robot design. Yet it is unclear whether trans, non-binary, or otherwise gender nonconforming people would view this as a positive and inclusive step, or as appropriative or otherwise problematic. In fact, little is known about trans and nonbinary perspectives on human-robot interaction, which have not been previously studied. In this work, we thus present the first study of trans and non-binary perspectives on robot design, with a particular focus on perceptions of robot gender and agender robot design. Our results suggest that trans and non-binary users readily accept robots depicted as agender, and view this as a positive design strategy that could help normalize non-cisgender identities. Yet our results also highlight key risks posed by this design strategy, including risks of backlash, caricature, and dehumanization, and show how those risks are shaped by political and economic factors.

Making Informed Decisions: Supporting Cobot Integration Considering Business and Worker Preferences

  • Dakota Sullivan
  • Nathan Thomas White
  • Andrew Schoen
  • Bilge Mutlu

Robots are ubiquitous in small-to-large-scale manufacturers. While collaborative robots (cobots) have significant potential in these settings due to their flexibility and ease of use, proper integration is critical to realize their full potential. Specifically, cobots need to be integrated in ways that utilize their strengths, improve manufacturing performance, and facilitate use in concert with human workers. Effective integration requires careful consideration and the knowledge of roboticists, manufacturing engineers, and business administrators. We propose an approach involving the stages of planning, analysis, development, and presentation, to inform manufacturers about cobot integration within their facilities prior to the integration process. We contextualize our approach in a case study with an SME collaborator and discuss insights learned.

Towards Collaborative Crash Cart Robots that Support Clinical Teamwork

  • Angelique Taylor
  • Tauhid Tanjim
  • Huajie Cao
  • Hee Rin Lee

Healthcare workers (HCWs) face many challenges during bedside care that impede team collaboration and often lead to poor patient outcomes. Robots have the potential to support medical decision-making, help identify medical errors, and deliver supplies to clinical teams in a timely manner. However, there is a lack of knowledge about using robots to support clinical team dynamics despite being used in surgery, healthcare operations, and other applications. To address this gap, we engaged in a co-design process of robots that support clinical teamwork. We collaboratively explore how robots can support clinical teamwork with HCWs. This collaborative process includes understanding the challenges they face during bedside care and envisioning robots that can help mitigate these issues. Our study shows that robots can act as a shared mental model for clinical teams, help close communication gaps, and provide procedural steps to assist HCWs with limited in-hospital experience. This research highlights new ways HRI researchers can deploy robots in acute care settings, as well as define appropriate levels of autonomy to maintain human control in safety-critical settings.

Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning

  • Tu Trinh
  • Haoyu Chen
  • Daniel S. Brown

We examine the problem of determining demonstration sufficiency: how can a robot self-assess whether it has received enough demonstrations from an expert to ensure a desired level of performance? To address this problem, we propose a novel self-assessment approach based on Bayesian inverse reinforcement learning and value-at-risk, enabling learning-from-demonstration (“LfD”) robots to compute high-confidence bounds on their performance and use these bounds to determine when they have a sufficient number of demonstrations. We propose and evaluate two definitions of sufficiency: (1) normalized expected value difference, which measures regret with respect to the human’s unobserved reward function, and (2) percent improvement over a baseline policy. We demonstrate how to formulate high-confidence bounds on both of these metrics. We evaluate our approach in simulation for both discrete and continuous state-space domains and illustrate the feasibility of developing a robotic system that can accurately evaluate demonstration sufficiency. We also show that the robot can utilize active learning in asking for demonstrations from specific states which results in fewer demos needed for the robot to still maintain high confidence in its policy. Finally, via a user study, we show that our approach successfully enables robots to perform at users’ desired performance levels, without needing too many or perfectly optimal demonstrations.

Fast Perception for Human-Robot Handovers with Legged Manipulators

  • Andreea Tulbure
  • Firas Abi-Farraj
  • Marco Hutter

Deploying perception modules for human-robot handovers is challenging because they require a high degree of reactivity, generalizability, and robustness to work reliably for a diversity of cases. Further complications arise as each object can be handed over in a variety of ways, causing occlusions and viewpoint changes. On legged robots, deployment is particularly challenging because of the limited computational resources and the image-space noise resulting from locomotion. In this paper, we introduce an efficient and object-agnostic real-time tracking framework, specifically designed for human-to-robot handover tasks with a legged manipulator. The proposed method combines optical flow with Siamese-network-based tracking and depth segmentation in an adaptive Kalman Filter framework. We show that we outperform the state-of-the-art for tracking during human-to-robot handovers with our legged manipulator. We demonstrate the generalizability, reactivity, and robustness of our system through experiments in different scenarios and by carrying out a user study. Additionally, as timing is proven to be more important than spatial accuracy for human-robot handovers, we show that we reach close to human timing performance during the approaching phase, both in terms of objective metrics and subjective feedback from the participants of our user study.

Workspace Optimization Techniques to Improve Prediction of Human Motion During Human-Robot Collaboration

  • Yi-Shiuan Tung
  • Matthew B. Luebbers
  • Alessandro Roncone
  • Bradley Hayes

Understanding human intentions is critical for safe and effective human-robot collaboration. While state of the art methods for human goal prediction utilize learned models to account for the uncertainty of human motion data, that data is inherently stochastic and high variance, hindering those models’ utility for interactions requiring coordination, including safety-critical or close-proximity tasks. Our key insight is that robot teammates can deliberately configure shared workspaces prior to interaction in order to reduce the variance in human motion, realizing classifier-agnostic improvements in goal prediction. In this work, we present an algorithmic approach for a robot to arrange physical objects and project “virtual obstacles” using augmented reality in shared human-robot workspaces, optimizing for human legibility over a given set of tasks. We compare our approach against other workspace arrangement strategies using two human-subjects studies, one in a virtual 2D navigation domain and the other in a live tabletop manipulation domain involving a robotic manipulator arm. We evaluate the accuracy of human motion prediction models learned from each condition, demonstrating that our workspace optimization technique with virtual obstacles leads to higher robot prediction accuracy using less training data.

When Do People Want an Explanation from a Robot?

  • Lennart Wachowiak
  • Andrew Fenn
  • Haris Kamran
  • Andrew Coles
  • Oya Celiktutan
  • Gerard Canal

Explanations are a critical topic in AI and robotics, and their importance in generating trust and allowing for successful human-robot interactions has been widely recognized. However, it is still an open question when and in what interaction contexts users most want an explanation from a robot. In our pre-registered study with 186 participants, we set out to identify a set of scenarios in which users show a strong need for explanations. Participants are shown 16 videos portraying seven distinct situation types, from successful human-robot interactions to robot errors and robot inabilities. Afterwards, they are asked to indicate if and how they wish the robot to communicate subsequent to the interaction in the video. The results provide a set of interactions, grounded in literature and verified empirically, in which people show the need for an explanation. Moreover, we can rank these scenarios by how strongly users think an explanation is necessary and find statistically significant differences. Comparing giving explanations with other possible response types, such as the robot apologizing or asking for help, we find that why-explanations are always among the two highest-rated responses, with the exception of when the robot simply acts normally and successfully. This stands in stark contrast to the other possible response types that are useful in a much more restricted set of situations. Lastly, we test for factors of an individual that might influence their response preferences, for example, their general attitude towards robots, but find no significant correlations. Our results can guide roboticists in designing more user-centered and transparent interactions and let explainability researchers develop more pinpointed explanations.

The Cyber-Physical Control Room: A Mixed Reality Interface for Mobile Robot Teleoperation and Human-Robot Teaming

  • Michael E. Walker
  • Maitrey Gramopadhye
  • Bryce Ikeda
  • Jack Burns
  • Daniel Szafir

In this work, we present the design and evaluation of an immersive Cyber-Physical Control Room interface for remote mobile robots that provides users with both robot-egocentric and robot-exocentric 3D perspectives. We evaluate the Cyber-Physical Control room against a traditional robot interface in a mock disaster response scenario that features a mixed human-robot field team. In our evaluation, we found that the Cyber-Physical Control Room improved robot operator effectiveness by 28% while navigating a complex warehouse environment and performing a visual search. The Cyber-Physical Control Room also enhanced various aspects of human-robot teaming, including social engagement, the ability of a remote robot teleoperator to track their human partner in the field, and opinions of human teammate leadership qualities.

Doodlebot: An Educational Robot for Creativity and AI Literacy

  • Randi Williams
  • Safinah Ali
  • Raúl Alcantara
  • Tasneem Burghleh
  • Sharifa Alghowinem
  • Cynthia Breazeal

Today, Artificial Intelligence (AI) is prevalent in everyday life, with emerging technologies like AI companions, autonomous vehicles, and AI art tools poised to significantly transform the future. The development of AI curricula that shows people how AI works and what they can do with it is a powerful way to prepare everyone, and especially young learners, for an increasingly AI-driven world. Educators often employ robotic toolkits in the classroom to boost engagement and learning. However, these platforms are generally unsuitable for young learners and learners without programming expertise. Moreover, these platforms often serve as either programmable artifacts or pedagogical agents, rarely capitalizing on the opportunity to support students in both capacities. We designed Doodlebot, a mobile social robot for hands-on AI education to address these gaps. Doodlebot is an effective tool for exploring AI with grade school (K-12) students, promoting their understanding of AI concepts such as perception, representation, reasoning and generation. We begin by elaborating Doodlebot’s design, highlighting its reliability, user-friendliness, and versatility. Then, we demonstrate Doodlebot’s versatility through example curricula about AI character design, autonomous robotics, and generative AI accessible to young learners. Finally, we share the results of a preliminary user study with elementary school youth where we found that the physical Doodlebot platform was as effective and user-friendly as the virtual version. This work offers insights into designing interactive educational robots that can inform future AI curricula and tools.

Anticipating the Use of Robots in Domestic Abuse: A Typology of Robot Facilitated Abuse to Support Risk Assessment and Mitigation in Human-Robot Interaction

  • Katie Winkle
  • Natasha Mulvihill

Domestic abuse research demonstrates that perpetrators are agile in finding new ways to coerce and to consolidate their control. They may leverage loved ones or cherished objects, and are increasingly exploiting and subverting what have become everyday ‘smart’ technologies. Robots sit at the intersection of these categories: they bring together multiple digital and assistive functionalities in a physical body, often explicitly designed to take on a social companionship role. We present a typology of robot facilitated abuse based on these unique affordances, designed to support systematic risk assessment, mitigation and design work. Whilst most obviously relevant to those designing robots for in-home deployment or intrafamilial interactions, the ability to coerce can be wielded by those who have any form of social power, such that our typology and associated design reflections may also be salient for the design of robots to be used in the school or workplace, between carers and the vulnerable, elderly and disabled and/or in institutions which facilitate intimate relations of care.

Dimensional Design of Emotive Sounds for Robots

  • Hannen Wolfe
  • Yiheng Su
  • Jue Wang

Non-Linguistic Utterances (NLUs) are essential parts of emotive exchanges, not only in human-human interactions but also in the context of human-robot interactions. This research aims to deepen our understanding of emotive sounds for the domain of human-robot exchanges. We investigated the connections between certain audio qualities and the perception of emotional arousal and pleasure, designing a novel mapping of musical and prosodic audio parameters to a dimensional model of emotion which allows a robot to express a range of emotions. To assess the emotive sounds, we conducted an end-user evaluation in which participants were asked to interpret the emotive NLUs conveyed by robots. In the evaluation we examined 4 archetypal emotions: excitement, contentment, sadness, and anger. We placed participants’ responses within the pleasure-arousal affect grid to analyze the distinctiveness of the emotive sounds. The study revealed that participants consistently associated excited, sad and angry NLUs with significantly different emotional states but did not do so for content NLUs. These findings contribute valuable insights into how to design NLUs which can enhance the emotional depth of human-robot interactions, with potential applications across various domains.

Imagination vs. Reality: Investigating the Acceptance and Preferred Anthropomorphism in Service HRI

  • Katharina Wzietek
  • Felix Wilhelm Siebert
  • Eileen Roesler

While the use of robots in public spaces is increasing, still few studies explore the resulting everyday human-robot interactions (HRI). The present study sought to bridge the disparity between real-world interactions and the frequently examined hypothetical interactions. To do so, we investigate the imagined and actual interaction with an ice cream serving robot. In two studies and an exploratory study comparison, we examined user acceptance and preference for the degree of anthropomorphic appearance. Although a typical human service task was taken over by a robot, an industrial robot was preferred according to participants’ ratings in both studies. Moreover, both studies demonstrated that robot enthusiasm significantly relates to participants’ acceptance of the robot for the task. Besides these commonalities, the results showed also that while humans were preferred over robots in the imagined setting, no clear preference was found in the real-life setting. Additional analyses compared the free text answers of the two studies and provided insights into participants’ general attitudes toward robots in the workforce. In line with the higher preferences for humans over robots in the imagined setting, considerably more participants mentioned a better customer experience with humans as important in the imagined study compared to the participants who interacted with the robot. The studies strikingly demonstrated that imaginary settings yield similar outcomes to those where participants physically engage with the robot in certain aspects, such as their preference for anthropomorphism. However, this phenomenon does not appear to hold for other facets, such as their favored service agent.

Affective and Cognitive Reactions to Robot-Initiated Social Control of Health Behaviors

  • Jiaxin Xu
  • Chao Zhang
  • Raymond H. Cuijpers
  • Wijnand A. IJsselsteijn

Health-related social control refers to intentional attempts to influence people’s health behaviors, often seen in personal relationships. Social robots hold promise in influencing people’s health by exerting health-related social control, but it is unclear which social control strategies used by robots are appropriate and potentially effective. This study investigates the effects of positive versus negative, and relationship-oriented versus target-oriented social control strategies from a social robot on people’s psychological reactions. In an online video prototype study, participants viewed scenarios of a social robot attempting to change their sedentary behaviors by using different strategies. We found that positive (versus negative) strategies elicited stronger positive affect, enjoyment, and perceived social appropriateness, reduced perceived threats to freedom, and strengthened behavioral intention. Meanwhile, the relationship-oriented (versus target-oriented) strategies elevated people’s negative affect, reduced enjoyment and perceived appropriateness, elevated perceived threats to freedom, and weakened behavioral intentions. Given these findings, we give recommendations for designing health influence strategies in social robots.

Enhancing Safety in Learning from Demonstration Algorithms via Control Barrier Function Shielding

  • Yue Yang
  • Letian Chen
  • Zulfiqar Zaidi
  • Sanne van Waveren
  • Arjun Krishna
  • Matthew Gombolay

Learning from Demonstration (LfD) is a powerful method for non-roboticists end-users to teach robots new tasks, enabling them to customize the robot behavior. However, modern LfD techniques do not explicitly synthesize safe robot behavior, which limits the deployability of these approaches in the real world. To enforce safety in LfD without relying on experts, we propose a new framework, SElding with Control barrier fUnctions in inverse REinforcement learning (SECURE), which learns a customized Control Barrier Function (CBF) from end-users that prevents robots from taking unsafe actions while imposing little interference with the task completion. We evaluate SECURE in three sets of experiments. First, we empirically validate SECURE learns a high-quality CBF from demonstrations and outperforms conventional LfD methods on simulated robotic and autonomous driving tasks with improvements on safety by up to 100%. Second, we demonstrate that roboticists can leverage SECURE to outperform conventional LfD approaches on a real-world knife-cutting, meal-preparation task by 12.5% in task completion while driving the number of safety violations to zero. Finally, we demonstrate in a user study that non-roboticists can use SECURE to effectively teach the robot safe policies that avoid collisions with the person and prevent coffee from spilling.

PoseTron: Enabling Close-Proximity Human-Robot Collaboration Through Multi-human Motion Prediction

  • Mohammad Samin Yasar
  • Md Mofijul Islam
  • Tariq Iqbal

As robots enter human workspaces, there is a crucial need for robots to understand and predict human motion to achieve safe and fluent human-robot collaboration (HRC). However, accurate prediction is challenging due to a lack of large-scale datasets for close-proximity HRC and the absence of generalizable algorithms. To overcome these challenges, we present INTERACT, a comprehensive multimodal dataset covering 3-D Skeleton, RGB+D, gaze, and robot joint data for human-human and human-robot collaboration. Additionally, we introduce PoseTron, a novel transformer-based architecture to address the gap in learning algorithms. PoseTron introduces a conditional attention mechanism in the encoder enabling efficient weighing of motion information from all agents to incorporate team dynamics. The decoder features a novel multimodal attention mechanism, which weights representations from different modalities and the encoder outputs to predict future motion. We extensively evaluated PoseTron by comparing its performance on the INTERACT dataset against state-of-the-art algorithms. The results suggest that PoseTron outperformed all other methods across all the scenarios, attaining lowest prediction errors. Furthermore, we conducted a comprehensive ablation study, emphasizing the importance of design choices, pointing towards a promising direction for integrating motion prediction with robot perception in safe and effective HRC.

Autonomy Acceptance Model (AAM): The Role of Autonomy and Risk in Security Robot Acceptance

  • Xin Ye
  • Wonse Jo
  • Arsha Ali
  • Samia Cornelius Bhatti
  • Connor Esterwood
  • Hana Andargie Kassie
  • Lionel Peter Robert

The rapid deployment of security robots across our society calls for further examination of their acceptance. This study explored human acceptance of security robots by theoretically extending the technology acceptance model to include the impact of autonomy and risk. To accomplish this, an online experiment involving 236 participants was conducted. Participants were randomly assigned to watch a video introducing a security robot operating at an autonomy level of low, moderate, or high, and presenting either a low or high risk to humans. This resulted in a 3 (autonomy) × 2 (risk) between-subjects design. The findings suggest that increased perceived usefulness, perceived ease of use, and trust enhance acceptance, while higher robot autonomy tends to decrease acceptance. Additionally, the physical risk associated with security robots moderates the relationship between autonomy and acceptance. Based on these results, this paper offer recommendations for future research on security robots.

Robots for Social Justice (R4SJ): Toward a More Equitable Practice of Human-Robot Interaction

  • Yifei Zhu
  • Ruchen Wen
  • Tom Williams

In this work, we present Robots for Social Justice (R4SJ): a framework for an equitable engineering practice of Human-Robot Interaction, grounded in the Engineering for Social Justice (E4SJ) framework for Engineering Education and intended to complement existing frameworks for guiding equitable HRI research. To understand the new insights this framework could provide to the field of HRI, we analyze the past decade of papers published at the ACM/IEEE International Conference on Human-Robot Interaction, and examine how well current HRI research aligns with the principles espoused in the E4SJ framework. Based on the gaps identified through this analysis, we make five concrete recommendations, and highlight key questions that can guide the introspection for engineers, designers, and researchers. We believe these considerations are a necessary step not only to ensure that our engineering education efforts encourage students to engage in equitable and societally beneficial engineering practices (the purpose of E4SJ), but also to ensure that the technical advances we present at conferences like HRI promise true advances to society, and not just to fellow researchers and engineers.

SESSION: Short Paper Contributions

Wrapyfi: A Python Wrapper for Integrating Robots, Sensors, and Applications across Multiple Middleware

  • Fares Abawi
  • Philipp Allgeuer
  • Di Fu
  • Stefan Wermter

Message oriented and robotics middleware play an important role in facilitating robot control, abstracting complex functionality, and unifying communication patterns between sensors and devices. However, using multiple middleware frameworks presents a challenge in integrating different robots within a single system. To address this challenge, we present Wrapyfi, a Python wrapper supporting multiple message oriented and robotics middleware, including ZeroMQ, YARP, ROS, and ROS 2. Wrapyfi also provides plugins for exchanging deep learning framework data, without additional encoding or preprocessing steps. Using Wrapyfi eases the development of scripts that run on multiple machines, thereby enabling cross-platform communication and workload distribution. We finally present the three communication schemes that form the cornerstone of Wrapyfi’s communication model, along with examples that demonstrate their applicability.

Dataset and Evaluation of Automatic Speech Recognition for Multi-lingual Intent Recognition on Social Robots

  • Antonio Andriella
  • Raquel Ros
  • Yoav Ellinson
  • Sharon Gannot
  • Séverin Lemaignan

While Automatic Speech Recognition (ASR) systems excel in controlled environments, challenges arise in robot-specific setups due to unique microphone requirements and added noise sources. In this paper, we create a dataset of initiating conversations with brief exchanges in 5 European languages, and we systematically evaluate current state-of-art ASR systems (Vosk, OpenWhisper, Google Speech and NVidia Riva). Besides standard metrics, we also look at two critical downstream tasks for human-robot verbal interaction: intent recognition rate and entity extraction, using the open-source Rasa chatbot. Overall, we found that open-source solutions as Vosk performs competitively with closed-source solutions while running on the edge, on a low compute budget (CPU only).

A Long-Range Mutual Gaze Detector for HRI

  • Simone Arreghini
  • Gabriele Abbate
  • Alessandro Giusti
  • Antonio Paolillo

The detection of mutual gaze in the context of human-robot interaction is crucial for the understanding of human partners’ behavior. Indeed, the monitoring of the users’ gaze from a long distance enables the prediction of their intention and allows the robot to be proactive. Nonetheless, current implementations struggle or cannot operate in scenarios where detection from long distances is required. In this work, we propose a ROS2 software pipeline that detects mutual gaze up to 5 m of distance. The code relies on robust off-the-shelf perception algorithms.

GARRY: The Gait Rehabilitation Robotic System

  • Benjamin O. Bestmann
  • Alex Chow
  • Alyssa Kubota
  • Laurel D. Riek

Gait rehabilitation is a critical aspect of post-stroke recovery, and emerging technologies such as virtual reality and wearables are playing a pivotal role in facilitating this process. However, despite the potential benefits, there is a significant gap in robot-based rehabilitative systems that facilitate repeated use by maintaining users’ attention long-term. Our research aims to bridge this gap by creating a comprehensive system that utilizes different feedback types and robotic assistance to support users’ gait rehabilitation outcomes. In this paper, we introduce GARRY (Gait Rehabilitation Robotic System), a new robotic system that provides interactive feedback during locomotor training. It promotes engagement by gamifying the rehabilitation process, offering a fun means for the user to meet their rehabilitation goals defined and set by physical therapists. GARRY also incorporates behavioral feedback to introduce a sense of companionship during a session. We make GARRY open-source to other researchers in hopes of encouraging accessibility and to promote research in the field. Our code can be found here:

Unlocking Human-Robot Dynamics: Introducing SenseCobot, a Novel Multimodal Dataset on Industry 4.0

  • Simone Borghi
  • Federica Zucchi
  • Elisa Prati
  • Andrea Ruo
  • Valeria Villani
  • Lorenzo Sabattini
  • Margherita Peruzzini

In the era of Industry 4.0, the importance of human-robot collaboration (HRC) in the advancement of modern manufacturing and automation is paramount. Understanding the intricate physiological responses of the operator when they interact with a cobot is essential, especially during programming tasks. To this aim, wearable sensors have become vital for real-time monitoring of worker well-being, stress, and cognitive load. This article presents an innovative dataset (SenseCobot) of physiological signals recorded during several collaborative robotics programming tasks. This dataset includes various measures like ElectroCardioGram (ECG), Galvanic Skin Response (GSR), ElectroDermal Activity (EDA), body temperature, accelerometer, ElectroEncephaloGram (EEG), Blood Volume Pulse (BVP), emotions and subjective responses from NASA-TLX questionnaires for a total of 21 participants. By sharing dataset details, collection methods, and task designs, this article aims to drive research in HRC advancing understanding of the User eXperience (UX) and fostering efficient, intuitive robotic systems. This could promote safer and more productive HRC amid technological shifts and help decipher intricate physiological signals in different scenarios.

REACT: Two Datasets for Analyzing Both Human Reactions and Evaluative Feedback to Robots Over Time

  • Kate Candon
  • Nicholas C. Georgiou
  • Helen Zhou
  • Sidney Richardson
  • Qiping Zhang
  • Brian Scassellati
  • Marynel Vázquez

Recent work in Human-Robot Interaction (HRI) has shown that robots can leverage implicit communicative signals from users to understand how they are being perceived during interactions. For example, these signals can be gaze patterns, facial expressions, or body motions that reflect internal human states. To facilitate future research in this direction, we contribute the \textttREACT database, a collection of two datasets of human-robot interactions that display users’ natural reactions to robots during a collaborative game and a photography scenario. Further, we analyze the datasets to show that interaction history is an important factor that can influence human reactions to robots. As a result, we believe that future models for interpreting implicit feedback in HRI should explicitly account for this history. \textttREACT opens up doors to this possibility in the future.

PedSUMO: Simulacra of Automated Vehicle-Pedestrian Interaction Using SUMO To Study Large-Scale Effects

  • Mark Colley
  • Julian Czymmeck
  • Mustafa Kücükkocak
  • Pascal Jansen
  • Enrico Rukzio

As automated vehicles become more widespread but lack a driver to communicate in uncertain situations, external communication, for example, via LEDs or displays, is evaluated. However, the concepts are mostly evaluated in simple scenarios, such as one person trying to cross in front of one automated vehicle. The traditional empirical approach fails to study the large-scale effects of these in this not-yet-real scenario. Therefore, we built PedSUMO, an enhancement to SUMO for the simulacra of automated vehicles’ effects on public traffic, specifically how pedestrian attributes affect their respect for automated vehicle priority at unprioritized crossings. We explain the algorithms used and the derived parameters relevant to the crossing. We open-source our code under and demonstrate an initial data collection and analysis of Ingolstadt, Germany.

Using 3D Mice to Control Robot Manipulators

  • Varad Dhat
  • Nick Walker
  • Maya Cakmak

Fluid 6DOF teleoperation of robot manipulators enables telemanipulation where autonomy is not possible, facilitates the collection of demonstration data, and aids routine robotics development. Amongst 6DOF input devices, 3D mice stand apart for their ergonomic design and low cost, but their sensitivity and users’ relative inexperience with them require special design considerations. We contribute a web software package that makes integrating 3D mice in robot manipulation interfaces easy. The package consists of configurable input signal processing schemes that can make the device more forgiving by, for instance, rejecting small inputs or emphasizing a dominant axis, and an interactive visual representation of the device’s 6DOF twist input, which helps with operator familiarization and provides a visual aide during teleoperation. We provide a demonstration interface illustrating a typical integration with a ROS/ROS2 robot system and give usage advice based on our research experience.

Understanding Fatigue Through Biosignals: A Comprehensive Dataset

  • Marta Gabbi
  • Luca Cornia
  • Valeria Villani
  • Lorenzo Sabattini

Fatigue is a multifaceted construct, that represents an important part of human experience. The two main aspects of fatigue are the mental one and the physical one, that often intertwine, intensifying their collective impact on daily life and overall well-being. To soften this impact, understanding and quantifying fatigue is crucial. Physiological data play a pivotal role in the comprehension of fatigue, allowing a precious insight into the level and type of fatigue experienced. Though the analysis of these biosignals, researchers can determine whether the person is feeling mental fatigue, physical fatigue or a combination of both. This paper introduces MePhy, a comprehensive dataset containing various biosignals, gathered while inducing different fatigue conditions, in particular mental and physical fatigue. Among the biosignals closely associated with stress situations, we chose: eye activity, cardiac activity, electrodermal activity (EDA) and electromyography (EMG). Data were collected using different devices, including a camera, a chest strap and different sensors from the BITalino kit.

UASOS: An Experimental Environment for Assessing Mental Fatigue & Cognitive Flexibility during Drone Operations

  • Marcel F. Hinss
  • Vincenzo Maria Vitale
  • Nhat Tien Phan
  • Raphaëlle N. Roy
  • Anke M. Brock

Mental fatigue from continuous operations without breaks represents a safety issue for military drone operations, as these systems are complex and operate during long shifts. Military operations are hard to study due to their sensitive nature. The open-access program UASOS serves as a testbed to examine the effects of mental fatigue in an ecologically valid environment. UASOS recreates fundamental aspects of military drone operations in a controllable environment that is easy enough for novices to understand but demanding enough to elicit mental fatigue. Participants alternate between navigating a drone-using either a trackball/mouse or a joystick-and searching for visual targets. The protocol is set up in a way that taxes the cognitive flexibility of participants by constantly requiring them to alternate between tasks. In addition, several parameters such as difficulty, duration, questionnaires, training phases, and more can be adapted. The task also allows for synchronization with physiological data using LabStreamingLayer. Implemented in python, the code is set up to be easily installed.

Sobotify: A Framework for Turning Robots into Social Robots

  • Heiko Hübert
  • Hae Seon Yun

Sobotify is a software framework, which aims at simplifying the process of using robots in the field of social robotics. This paper delineates the design and usage of the framework. With Sobotify, even non-technical people should be enabled to use robots for their specific purposes, such as teachers in a classroom, therapists in a physiological or psychological therapy or childcare workers in kindergarten. During the development process of Sobotify, feedback from teachers at a vocational school have been taken into account in order to adjust the tools to their needs. The framework is designed to work with different robots including both humanoid (NAO and Pepper) and non-humanoid robots such as toy robots (Cozmo) with advanced abilities as well as very simple toy robots (MyKeepon). The framework was tested by two research works which proved that Sobotify is applicable in different setups. Further development is already planned for the next months, e.g. integration of additional robots and extension of tools.

Evaluation Tools for Human-AI Interactions Involving Older Adults with Mild Cognitive Impairments

  • Daisy M. Kiyemba
  • Jasmin Marward
  • Elizabeth J. Carter
  • Adam Norton

As artificial intelligence (AI) systems have already proven useful in human lives generally, there is an opportunity for specialized human-AI interaction (HAI) systems to support and provide care for older adults with mild cognitive impairment (MCI). However, the integration of this technology in this population must be thoughtfully designed to accommodate specific needs and limitations. This includes careful measurement of both humans and systems. We developed an evolving dataset categorizing relevant measurement tools into five groups: cognitive ability, demographics & personality, activity level, state of mind, and perceptions of the AI system. Each instance of the tool being used in the literature cataloged in the dataset is qualified in terms of how likely we would recommend using it in the domain of HAI for older adults with MCI based on contextual factors and internal reliability measures. This dataset will serve as a valuable resource for future research, aiding in the identification of promising areas and trends in AI systems for older adults with MCI as well as providing essential tools for future studies.

Probabilistic Fusion of Persons’ Body Features: The Mr. Potato Algorithm

  • Séverin Lemaignan
  • Lorenzo Ferrini

Multi-modal social perception usually involves several independent software modules, detecting for instance faces, voices, body skeletons. Those features need then to be matched to each other, to create a complete model of a person. While the problem is simple in one-to-one interactions, multi-party interactions require to optimize a probabilistic graph in order to find the most likely persons–features associations, while ensuring practical properties like stability over time. This paper presents an open-source algorithm that searches over all possible partitions of the relationship graph to identify the best partition. We playfully call this algorithm Mr. Potato, after the eponymous children’ game.

RW4T Dataset: Data of Human-Robot Behavior and Cognitive States in Simulated Disaster Response Tasks

  • Liubove Orlov Savko
  • Zhiqin Qian
  • Gregory Gremillion
  • Catherine Neubauer
  • Jonroy Canady
  • Vaibhav Unhelkar
  • Catherine Neubauer

To forge effective collaborations with humans, robots require the capacity to understand and predict the behaviors of their human counterparts. There is a growing body of computational research on human modeling for human-robot interaction (HRI). However, a key bottleneck in conducting this research is the relative lack of data of cognitive states — like intent, workload, and trust — which undeniably affect human behavior. Despite their significance, these states are elusive to measure, making the assembly of datasets a challenge and hindering the progression of human modeling techniques. To help address this, we first introduce Rescue World for Teams (RW4T): a configurable testbed to simulate disaster response scenarios requiring human-robot collaboration. Next, using RW4T, we curate a multimodal dataset of human-robot behavior and cognitive states in dyadic human-robot collaboration. This RW4T dataset includes state, action and reward sequences, and all the necessary data to replay a visual task execution. It further contains psychophysiological metrics like heart rate and pupillometry, complemented by self-reported cognitive state measures. With data from 20 participants, each undertaking five human-robot collaborative tasks, this dataset (comprising of 100 unique trajectories) accompanied with the simulator can serve as a valuable benchmark for human behavior modeling.

A Lightweight Artificial Cognition Model for Socio-Affective Human-Robot Interaction

  • Andrés A. Ramirez Duque
  • Alan Lindsay
  • Mary Ellen Foster
  • Ronald P. A. Petrick

The software submission presents a fully working artificial cognition model, which controls a NAO social robot. The model was specifically designed to control a socio-affective companion robot for use in a medical setting. It was deployed using embedded hardware: a Raspberry Pi 4B and a Jetson Nano Board, and an external RGB-D camera. Based on the ROS operating system, this software package includes components for social signal processing, behaviour selection, affective behaviour rendering, and a web-based user interface. The robot’s behaviours are selected by a planning system, which generates the robot’s behaviours based on the state of the interaction, the progress of the medical procedure, and the user’s affective state. The system has been tested in simulated environments and is currently being used in two clinics to perform a usability test and will subsequently be used to carry out a series of clinical trials

Benchmark Movement Data Set for Trust Assessment in Human Robot Collaboration

  • Matthias Rehm
  • Kasper Hald
  • Ioannis Pontikis

Trust is a factor that is becoming more prominent in human robot interaction research. Only few approaches so far tackle the challenge of data-driven trust assessment. In this paper, we present a data set consisting of motion tracking data from an industrial human robot collaboration task. The data is collected during a trust manipulation experiment that has been designed to elicit different trust levels in the participants. Additionally, participants filled out a standard trust questionnaire. The data set allows for developing and testing data-driven trust assessment algorithms.

Moving Horizon Planning for Human-Robot Interaction

  • Heiko Renz
  • Maximilian Krämer
  • Torsten Bertram

The collaboration and interaction between humans and robots intensify with ongoing research and industry needs. Robots require a motion planner that contributes to a safe environment for humans. This paper provides the online trajectory planner Moving Horizon Planning for Human-Robot Interaction (MHP4HRI), customizable for various robots, considering obstacles and humans in their environment. The planner generates motion commands in a moving horizon manner, similar to Model Predictive Control. This enables robots to react to dynamic changes in the environment in real-time. Descriptions of the planner and the underlying algorithms are given, as well as details about the provided framework regarding the benefits and usage for the community. Furthermore, we aim to provide a growing framework with new features in the future regarding the optimization and interaction with the environment, especially humans. The code, implemented mainly in C++ for the Robot Operating System (ROS), is available at GitHub:

OpenVP: A Customizable Visual Programming Environment for Robotics Applications

  • Andrew Schoen
  • Bilge Mutlu

Authored robotics applications have a diverse set of requirements for their authoring interfaces, being dependent on the underlying architecture of the program, the capabilities of the programmers and engineers using them, and the capabilities of the robot. Visual programming approaches have long been favored for both novice-level accessibility and clear graphical representations, but current tools are limited in their customizability and ability to be integrated holistically into larger design interfaces. OpenVP attempts to address this by providing a highly configurable and customizable component library that can be integrated easily into other modern web-based applications.

Towards Reproducible Language-Based HRI Experiments: Open-Sourcing a Generalized Choregraphe Project

  • Uthman Tijani
  • Hong Wang
  • Zhao Han

We are witnessing increasing calls for reproducibility and replicability in HRI studies to improve reliability and confidence in empirical findings. One solution to facilitate this is using a robot platform that researchers frequently use, making it easier to replicate studies to verify results. In this work, we focus on a popular, affordable, and rich-in-functionality robot platform, NAO/Pepper, and contribute a generalized experiment project specifically for conducting language-based HRI experiments where a robot instructs a human for a task, including objective data collection.

Specifically, we first describe a concrete workflow from an existing experiment and how it is generalized. We then evaluate the generalized project with a case study to show how adopters can quickly adapt to their specific experiment needs. This work provides inspiration for HRI researchers to not only provide their experiment code as supplementary material but also generalize them to benefit other researchers to advance empirical research in HRI. The generalized Choregraphe project with documentation, demo, and usage notes is available under MIT license on GitHub at We welcome questions by posting GitHub issues and pull requests to share adapted packages.