One way to unify the scope of current HRI research is to condense the five dimensions of designer influence into a single concept as exemplified in our proposed scale of interaction, Figure 3, with the caveat that this single concept cannot capture every nuance and possible design of every HRI problem. The concept of dynamic interaction seems to capture the current research direction of many HRI efforts.

Dynamic interaction includes time- and task-varying changes in autonomy, information exchange, team organization and authority, and training. It applies to both remote and proximate interactions, including social and physical interactions. By including variable autonomy assignments, the concept of dynamic interaction subsumes adaptive and dynamic autonomy as a special case [78, 219-224]. By including information exchange, dynamic interaction includes adaptive and adaptable interfaces [28, 78, 225]. By including team organization and authority, mixed initiative interaction [123, 158, 226, 227] is addressed. By including training, interactive learning is included.

More importantly, the concept of dynamic interaction places the emphasis on shaping the types of interactions that can and will emerge as humans and robots interact. The scope of HRI research and design, therefore, includes all efforts at evaluating systems and interaction paradigms, designing autonomy algorithms in the context of HRI, designing interfaces and information exchange protocols, defining and switching roles, and influencing learning and training. This emphasis on dynamic interaction differs sharply from the historically static interactions of pure teleoperation and pure supervisory control.

Note that some current research efforts and methods do not naturally fit into the dynamic interaction framework. These include several aspects of task shaping, including ethnographic studies, goal-directed task analyses, and some cognitive science-based work. However, understanding existing processes and potential use patterns helps researchers better understand the fluid interaction patterns that are likely to exist in practice, and then design interactions that support, improve, and extend these interaction patterns.

There are formal processes for understanding how the task should be done and is currently done. These processes include goal-directed task analyses, cognitive work analyses, and ethnographic studies[105, 193, 215]. Although frequently used to specify how a task is done and how it should be done, it is imperative to consider how the task will be done, including unintended consequences of design [216, 217].

One reason that little is written about task-shaping is because designers are implicitly trying to create technology and interactions that accomplish some task or function. Indeed, Woods has persuasively argued that designing a system is equivalent to making a hypothesis about how the artifact will positively shape the experience [218]. Nevertheless, it is important to consider how the task might be modified to better support interaction. Examples of explicit task-shaping include designing space or underwater equipment and tools so that handles and connectors can be manipulated by a robotic arm, “pre-cleaning” a room so that a robot vacuum can accomplish its task most efficiently [170], and performing pre-inspection tasks used to form maps and plans that can be executed by a robot.

On one hand, it is important to minimize the amount of human training and adaptation required to interact with robots that are used in therapeutic or educational roles for children, autistic individuals, or mentally challenged individuals. On the other hand, it is important that HRI include proper training for problems that include, for example, handling hazardous materials; similarly the goal of using robots in therapeutic and educational roles implies that humans should adapt and learn in response to interaction [168] . In this section, we discuss not only HRI domains that require minimum operator training, but also domains that require careful training. We also discuss efforts aimed to train HRI scientists and designers, and then conclude with a discussion of how the concept of training can be used to help robots evolve new skills in new application domains.

Minimizing Operator Training Minimizing training appears to be an implicit goal for “edutainment” robots, which include robots designed for use in classrooms and museums, for personal entertainment, and for home use. These robots are typically designed to be manageable by a wide variety of humans, and training can range from instruction manuals, instruction from a researcher, or instructions from the robot itself [128, 169].

One relevant study explored how ROOMBA robots are used in practice without attempting to make operators use the robots in a specific way [170]. Such studies are important because they can be used to create training materials that guide expectations and alert humans to possible dangers. Other such studies include those that explore how children use education robots in classroom settings [168], investigate how disabled children interact with robots in social settings [30], support humans in the house [171], and identify interaction patterns with museum guide robots [169].

Complementing such studies are efforts to use archetype patterns of behavior and well-known metaphors that trigger correct mental models of robot operation. Examples include the often stated hypotheses that people with “gaming experience” will be able to interact better (in some sense) with mobile robots than those with limited experiences in games [172]. We are not aware of any studies that directly support this hypothesis, but if it is true then it would seem to suggest that people with experiences in video-conferencing, instant-messaging, and other computer-mediated forms of communication might more naturally interact with robots. Whether this hypothesis is true is a matter of future work, but it is almost certainly true that such experiences help people form mental models that influence interactions [173]. Designers are seeking (a) to identify interaction modes that invoke commonly held mental models [174] such as those invoked by anthropomorphic robots [175] or (b) to exploit fundamental cognitive, social, and emotional processes [176]. One possible caution for these efforts is that robots may reach an “uncanny valley” where expectations evoked by the robot fall short of actual behavior producing an interaction that can feel strangely uncomfortable to humans [88, 177]. However, this uncanny valley theory is unproven although researchers are now trying to experimentally verify its existence [178].

Efforts to Train Humans. In contrast to the goal of minimizing training in edutainment robots, some application domains involving remote robots require careful training because operator workload or risk is so high. Important examples of such training are found in military and police applications, space applications, and in search and rescue applications. Training for military and police applications is typified by “bomb squad” robots, training for space applications is typified by telemanipulation tasks [179], and training for military and civilian search and rescue is typified by reconnaissance using small, “human-packable robots” [180]. In both the military and search application domains, training efforts exist for both air and ground robots, and these efforts tend to emphasize the use of mobile robots in a mission context [51]. Training efforts include instructions on using the interface, interpreting video, controlling the robot, coordinating with other members of the team, and staying safe while operating the robot in a hostile environment. Such training is often given to people who are already experts in their fields (such as in search and rescue), but is also given to people who may be relatively inexperienced. In the military, police and space domains, training programs may be complemented by selection criteria to help determine individuals are likely to be better (in some sense) at managing a robot [181]. Selection appears to have received more attention in air robots than ground robots.

By contrast to interactions with remote robots, many applications involving proximate robots are designed to produce learning or behavioral responses with humans. Therapeutic and social robots are designed to change, educate, or train people, especially in long-term interactions [168, 182, 183]. People also adapt to service robots over the long-term and over a wide range of tasks [184], and there is growing evidence that many long-term interactions require mutual adaptation including with human bystanders [185-187]. Importantly, culture appears to influence both long-term and short-term adaptation, at least as far as accepting interactions with a robot [60, 142, 188-191].

Training Designers. Importantly, an often overlooked area is the training of HRI researchers and designers in the procedures and practices of those whom they seek to help. Important examples of training researchers include Murphy’s workshops on search and rescue robotics [192], tutorials and workshops on methodologies for understanding a work-practice domain and field studies [193], tutorials for young researchers on search and rescue [42, 43],and tutorials and workshops on metrics or experiment design for robot applications [194].

Training Robots. It is tempting to restrict training to the education of the human side of HRI, but this would be a mistake given current HRI research. In HRI, robots are also learning, both offline as part of the design process [31, 195] and online as part of interaction, especially long-term interaction [127, 196]. Such learning includes improving perceptual capabilities through efficient communication between humans and robots [127, 196-198], improving reasoning and planning capabilities through interaction [199, 200], and improving autonomous capabilities [201]. Approaches to robot learning include teaching or programming by demonstration [202-207], task learning [127, 195, 200], and skill learning including social, cognitive and locomotion skills [136, 199, 208-210]. Some researchers are exploring biologically inspired learning models, including how teaching among humans or social animals can be used to train a robot [208, 211]; others are exploring how learning can become more efficient if it leverages information about how the human brain learns in very few trials [212].

Interestingly, it can be argued that providing support for efficient programming or knowledge management systems is an important aspect of training robots in HRI [39, 130]. Additionally, it can be argued that sensitizing a robot to issues of culture and etiquette allows them to adapt to slowly changing humhuman norms of behavior [22, 213, 214].

A question that has received considerable attention, but which is directly addressed by few scientific studies, is how many remote robots a single human can manage. In general, the answer is dependent on factors such as the level of autonomy of the robot (teleoperation requires vastly more direct attention from the human), the task (which defines the type and quantity of data being returned to the human and the amount of attention and cognitive load required of the human), and the available modes of communication.

In the search and rescue domain, Murphy [144] asserts that the demands of the task, the form factor of the robot, and the need to protect robot operators requires at least two operators, an observation that has received strong support from field trials using mature technologies [148], and partial support in search and rescue competitions using less mature but more ambitious technologies [145]. In other domains, some assert that, given sophisticated enough autonomy and possibly coordinated control, it is possible for a single human to manage more than one robotic asset [149, 150] though the task may still need another human to interpret sensor information. Still others assert that this problem is ill-formed when robots are used primarily as an information-gathering tool [151]. An intermediate position is that the right question should not focus on how many robots can be managed by a single human, but rather the following: how many humans does it take to efficiently manage a fixed number of robots, allowing for the possibility of adaptable autonomy and dynamic handoffs between humans [152].

One measure that has received some attention in the literature is the notion of fan-out, which represents an upper bound on the number of independent, homogeneous robots that a single person can manage [153, 154]. This measure is supported by a limited set of techniques for estimating it [69]. Some work has been done to refine the fan-out to apply to teams of heterogeneous robots [155] and to tighten the bound by identifying various aspects of interaction [150]. In its present form, however, it is clear that fan-out is only a designer guideline and is insufficient, for example, to provide a trigger strategy [78] for adaptive automation. Alternatives to fan-out include predicting the performance of a team of heterogeneous robots from measurements of neglect tolerance and interaction times [156].

In addition to the number of humans and robots in a team, a key problem is the organization of the team [157, 158]. One important organizational question is who has authority to make certain decisions: robot, interface software, or human? Another important question is who has authority to issue instructions or commands to the robot and at what level: strategic, tactical, or operational? A third important question is how conflicts are resolved, especially when robots are placed in peer-like relationships with multiple humans. A fourth question is how roles are defined and supported: is the robot a peer, an assistant, or a slave; does it report to another robot, to a human, or is it fully independent?

Spanning all of these questions is whether the organizational structure is static or dynamic, with changes in responsibilities, authorities, and roles. In one study, managing multiple robots in a search and rescue domain under either manual or coordinated control produced results that strongly favored coordinated control [159]. In another study, four autonomy configurations, including two variations of sliding autonomy, were managed by a human working on a construction task with a team of heterogeneous robots [152]. In this study, the tradeoffs between time to completion, quality of behavior, and operator workload were strongly evident. This result emphasizes the importance of using dynamic autonomy when the world is complex and varies over time. In a third study, researchers explored how making coordination between robots explicit can reduce failures and improve consistency, in contrast to traditional interfaces [160]. In a fourth study, researchers explored the minimal amount of gestural information required to command various formations to a team of robots[161].

In many existing and envisioned problems, HRI will include not only humans and robots interacting with each other, but also coordinating with software agents. The most simple form of this is a three-agent problem which occurs when an intelligent interface is the intermediary between a human and a remote robot [162]. In this problem, the interface agent can monitor and categorize human behavior, monitor and detect problems with the robot, and support the human when workload levels, environment conditions, and robot capabilities change. A more complicated form of this teaming is in anticipated NASA applications where multiple distributed humans will interact with robots and with software agents that coordinate mission plans, human activities, and system resources [163].

A final issue that is starting to gain attention is the role of the human [164]. While much of the discussion up to this point is with respect to humans and robots performing a task together, there are cases where the robot may have to interact with bystanders or with people who are not expecting to work with a robot. Examples include the urban search and rescue robot that comes across a human to be rescued, a military robot in an urban environment that must interact with civilians, and a health assistant robot that must help a patient and interact with visitors. The role of the robot with respect to humans must be taken into account. The role of the human will be discussed in more detail in Section IV.

haptic robot interaction from Georgia Tech [102],
a “physical icon” for flying a UAV from Brigham Young University,
peer-to-peer interaction with the robot Kaspar from the University of Hertfordshire [298],
teleoperation of NASA’s Robonaut [205], a PDA-based interface for flying a UAV from Brigham Young University,
gesture- and speech-based interaction with MIT’s Leonardo [189],
a touchscreen interaction with a Cogniron robot [169],
(center) physical interaction with the RI-MAN robot [24].
(All images used with permission.)

Autonomy is only one of the components required to make an interaction beneficial. A second component is the manner in which information is exchanged between the human and the robot. Efficient interactions are characterized by productive exchanges between the human and robot. Measures of the efficiency of an interaction include the interaction time required for intent and/or instructions to be communicated to the robot [69], the cognitive or mental workload of an interaction [34], the amount of situation awareness produced by the interaction [105] (or reduced because of interruptions from the robot), and the amount of shared understanding or common ground between humans and robots[106, 107] .

There are two primary dimensions that determine the way information is exchanged between a human and a robot: the communications medium and the format of the communications. The primary media are delineated by three of the five senses: seeing, hearing, and touch. These media are manifested in HRI as follows:

• visual displays, typically presented as graphical user interfaces or augmented reality interfaces [108-111],
• gestures, including hand and facial movements and by movement-based signaling of intent [29, 112-114],
• speech and natural language, which includes both auditory speech and text-based responses, and which frequently emphasizes dialog and mixed-initiative interaction [115, 116],
• non-speech audio, frequently used in alerting [117], and
• physical interaction and haptics, frequently used remotely in augmented reality or in teleoperation to invoke a sense of presence especially in telemanipulation tasks [12, 118] and frequently used proximately to promote emotional, social, and assistive exchanges [119-122].

Recently, attention has focused on building multimodal interfaces [123], partly motivated by a quest to reduce workload in accordance to Wickens’ multiple resource theory [36] and partly motivated by a desire to make interactions more natural and easier to learn [124-126].

The format of the information exchange varies widely across domains. Speech- and natural language-based exchanges can be scripted and based on a formal language, can attempt to support full natural language, or can restrict natural language to a subset of language and a restricted domain (see, for example, [52, 127-130]). Importantly, speech-based exchanges must not only address the content of information exchanged, but also the rules of such exchange a lá the Gricean maxims [131], which ask to what extent the speech is truthful, relevant, clear and informative. Haptic information presentation can include giving warnings through vibrations, promoting the feeling of telepresence, supporting spatial awareness through haptic vests, and communicating specific pieces of information through haptic icons (see, for example, [132-134]). Audio information presentation can include auditory alerts, speech-based information exchange, and 3D awareness (see, for example, [32]). Presenting social information can include attentional cueing, gestures, sharing physical space, imitation, sounds, facial expression, speech and natural language [135-142]. Finally, graphical user interfaces present information in ways that include ecological displays, immersive virtual reality, and traditional windows-type interactions (see, for example, [27, 108, 109, 143]).

Figure 2. Representative types of robots. In clockwise order beginning in the upper left:

RepileeQ2 — an extremely sophisticated humanoid [136];
Robota — humanoid robots as “educational toys” [21];
SonyAIBO — a popular robot dog ;
(below the AIBO) A sophisticated unmanned underwater vehicle [176];
Shakey — one of the first modern robots, courtesy of SRI International, Menlo Park, CA [279];
Kismet — an anthropomorphic robot with exaggerated emotion [65];
Raven — a mini-UAV used by the US military [186];
iCAT — an emotive robot [Phillips Research, The Netherlands];
iRobot (registered TM) PackBot (registered TM) — a robust ground robot used in military applications [135].
(All images used with permission.)

Autonomy is not an end in itself in the field of HRI, but rather a means to supporting productive interaction. Indeed, autonomy is only useful insofar as it supports beneficial interaction between a human and a robot. Consequently, the physical embodiment and type of autonomy varies dramatically across robot platforms; see Figure 2, which shows a cross section of the very many different types of physical robots.

Perhaps the most strongly human-centered application of the concept of autonomy is in the notion of level of autonomy (LOA). Levels of autonomy describe to what degree the robot can act on its own accord. Although many descriptions of LOA have been seen in the literature, the best and most widely-cited description is by Tom Sheridan [77]. In Sheridan’s scale, there is a continuum from the entity being completely controlled by a human (i.e. teleoperated), through the entity being completely autonomous and not requiring input or approval of its actions from a human before taking actions:

1. Computer offers no assistance; human does it all
2. Computer offers a complete set of action alternatives
3. Computer narrows the selection down to a few choices
4. Computer suggests a single action
5. Computer executes that action if human approves
6. Computer allows the human limited time to veto before automatic execution
7. Computer executes automatically then necessarily informs the human
8. Computer informs human after automatic execution only if human asks
9. Computer informs human after automatic execution only if it decides to
10. Computer decides everything and acts autonomously, ignoring the human

Variations of this scale have been developed and used by various authors [78, 79]. Importantly, Miller and Parasuraman have noted that such scales may not be applicable to an entire problem domain but are rather most useful when applied to each subtask within a problem domain [80]. The authors further suggest that previous scales actually represent an average over all tasks.

While such (average) scales are appropriate to describe how autonomous a robot is, from a human-robot interaction point of view, a better way to consider autonomy is by describing to what level the human and robot interact and the degree to which each is capable of autonomy.

Figure 3. Levels of autonomy with emphasis on human interaction.

The scale presented in Figure 3 gives an emphasis to mixed-initiative interaction, which has been defined as a “flexible interaction strategy in which each agent (human and [robot]) contributes what it is best suited at the most appropriate time” [81]. Various and different HRI issues arise along this scale. On the direct control side, the issues tend toward making a user interface that reduces the cognitive load of the operator. On the other extreme of peer-to-peer collaboration, issues arise in how to create robots with the appropriate cognitive skills to interact naturally or efficiently with a human.

Matt Johnson — collaborative control and coactive design

Note that in order for the robot to achieve peer-to-peer collaboration, it must indeed be able to flexibly exhibit “full autonomy” at appropriate times. Moreover, it may need to support social interactions. As a result, peer-to-peer collaboration may be considered more difficult to achieve than full autonomy.

Autonomy is implemented using techniques from control theory, artificial intelligence, signal processing, cognitive science, and linguistics. A common autonomy approach is sometimes referred to as the sense-plan-act model of decision-making [15]. This model has been a target of criticism [82] and sometimes rightfully so, but much of the criticism may be a function of the early capacities of robots such as Shakey [14] rather than failings of the model per se. This model is typified by artificial intelligence techniques, such as logics and planning algorithms [83]. The model can also incorporate control theoretic concepts, which have been used very successfully in aviation, aeronautics, missile control, and etc. (see, for example, [84]).

In the mid-1980s, Brooks, Arkin, and others revolutionized the field of robotics by introducing a new autonomy paradigm that came to be known as behavior-based robotics. In this paradigm, behavior is generated from a set of carefully designed autonomy modules that are then integrated to create an emergent system [17, 18, 85]. These modules generate reactive behaviors that map sensors directly to actions, sometimes with no intervening internal representations. This model for behavior generation was accompanied by hardware development that allowed autonomy modules to be implemented in the small form factors required for many robotics applications.

Today, many researchers build sense-think-act models on top of a behavior-based substrate to create hybrid architectures [15]. In these systems, the low-level reactivity is separated
from higher level reasoning about plans and goals [86]. Some have developed mathematics and frameworks that can be viewed as formalizations of hybrid architectures and which are referred to as theories of intelligent control [63, 87]. Interestingly, some of the most challenging problems in developing (hybrid) behaviors is in producing natural and robust activity for a humanoid robot [19, 33, 88].

Complementing the advancement of robotic control algorithms has been the advancement of sensors, sensor-processing, and reasoning algorithms. This is best represented by the success of the field of probabilistic robotics, typified by probabilistic algorithms for localization and mapping [89, 90]. It is no overstatement to say that these algorithms, which frequently exploit data from laser and other range finder devices, have allowed autonomy to become truly useful for mobile robots [91], especially those that require remote interaction through periods of autonomous behavior and autonomous path planning [52, 92-95]. Although probabilistic algorithms can be computationally expensive, the memory capacity, computational speed, and form factor of modern computers have allowed these algorithms to be deployable.

The areas of representing knowledge and performing reasoning, especially in team contexts, have also grown. Example developments include the emergence of belief-desire-intention architectures [96], joint intention theory [97], affect-based computing [16, 29, 98], and temporal logics.

  Lanny Lin made contributions in March 2012.