Beyond Failure Recovery: An Engagement-Aware Human-in-the-loop Framework for Robotic Systems


Jiaying Fang, Joyce Yang, Zhanxin Wu, Bohan Yang, Tapomayukh Bhattacharjee

Paper ID 118

Session HRI

Poster session details TBA

Abstract: Conventional human-in-the-loop approaches typically involve users only when a robot encounters failure or uncertainty, treating humans primarily as tools to improve robot performance. However, in many human-centered robotics settings, interaction should support user engagement, keeping users meaningfully involved in decision-making rather than limiting them to failure-driven interventions. For many users, this cannot be achieved through limited, failure-driven interaction alone; they wish to remain involved in the robot’s decision-making to sustain engagement throughout the task. This is particularly compelling in physical caregiving, where mobility limitations can reduce users’ ability to intervene or modulate the robot’s behavior in the moment. As a result, interaction policies that engage users only upon failure may further reduce engagement by relegating users to passive observers for long stretches of the task. For example, a user with mobility limitations may experience reduced engagement when being continuously and passively fed by a robot. At the same time, overly frequent interaction can be tiring and increase the user’s workload.
    To address this trade-off, we propose Engagement-aware MPC (E-MPC), a user-engagement-aware method that plans interaction to maintain engagement while respecting a workload constraint. E-MPC leverages a user interaction dynamics model that captures how user engagement evolves as a function of both the frequency and type of interaction. Rather than requesting input only when difficulties arise during task execution, the robot proactively considers the user’s preferred level of engagement throughout the task, balancing autonomy and interaction while ensuring task success. We evaluate E-MPC in simulation with several ablations and baseline comparisons. Baselines optimize for task success alone or jointly for user workload and task success. Results demonstrate the effectiveness of our approach across diverse user personas. In addition, we conduct a real-world user study with participants with emulated mobility limitations on a robot-assisted bite acquisition system, showing that E-MPC improves user experience while maintaining task success.