Beyond Failure Recovery: An Engagement-Aware Human-in-the-loop Framework for Robotic Systems

Jiaying Fang, Joyce Yang, Zhanxin Wu, Bohan Yang, Tapomayukh Bhattacharjee

Cornell University

RSS 2026

Conventional human-in-the-loop approaches typically involve users only when a robot encounters failure or uncertainty, treating humans primarily as tools to improve robot performance. However, in many human-centered robotics settings, interaction should support user engagement, keeping users meaningfully involved in decision-making rather than limiting them to failure-driven interventions.

To address this trade-off, we propose Engagement-aware MPC (E-MPC), a user-engagement-aware method that plans interaction to maintain engagement while respecting a workload constraint. E-MPC leverages a user interaction dynamics model that captures how user engagement evolves as a function of both the frequency and type of interaction. Rather than requesting input only when difficulties arise during task execution, the robot proactively considers the user's preferred level of engagement throughout the task, balancing autonomy and interaction while ensuring task success.

Approach

E-MPC selects queries that drive engagement g_t toward the user's target g_des while keeping workload w_t bounded. When task success is at risk, it requires a task-assistive query; otherwise it may pick an engagement-oriented query or proceed autonomously. The controller replans in a receding-horizon manner.

Simulation Experiments

We compare E-MPC against four baselines — AlwaysQuery, NeverQuery, TaskConfidenceAware, and WorkloadAware (state-of-the-art) — on Skill Success, Engagement Tracking Accuracy, and Satisfaction, across three personas (g_des ∈ {0.2, 0.5, 0.8}).

Simulation results across personas and workload thresholds.

E-MPC tracks engagement accurately for every persona and outperforms all baselines on the Satisfaction metric.

Effect of fake queries and comparison with Random / Periodic baselines.

Adding engagement-oriented ("fake") MCQ queries yields statistically significant satisfaction gains, and E-MPC also beats Random and Periodic query schedulers that ignore task uncertainty and user state.

Real-Robot User Study

We ran an IRB-approved study on a Kinova Gen3 bite acquisition system with 10 participants (ages 19–29) wearing resistance bands to emulate mobility limitations.

User study setup with emulated mobility limitations.

Compared to the WorkloadAware baseline, E-MPC significantly improves Engagement Tracking Accuracy, Interaction Satisfaction, and Agency Satisfaction, while matching it on Task Success and workload compliance. 9/10 participants preferred E-MPC.

User study results across four subjective metrics.

BibTeX

@inproceedings{fang2026empc,
    title={Beyond Failure Recovery: An Engagement-Aware Human-in-the-loop Framework for Robotic Systems},
    author={Fang, Jiaying and Yang, Joyce and Wu, Zhanxin and Yang, Bohan and Bhattacharjee, Tapomayukh},
    booktitle={Proceedings of Robotics: Science and Systems (RSS)},
    year={2026}
}

Acknowledgments

This work was partly funded by National Science Foundation IIS #2132846, and CAREER #2238792.