A Human-in-the-Loop Confidence-Aware Failure Recovery Framework for Modular Robot Policies

Rohan Banerjee1, Krishna Palempalli*1, Bohan Yang*1, Jiaying Fang1, Alif Abdullah1, Tom Silver2, Sarah Dean†1, Tapomayukh Bhattacharjee†1
1Cornell University, 2Princeton University (Work done at Cornell University)

HRI 2026

Paper + Appendix Video Code

Abstract

Robots operating in unstructured human environments inevitably encounter failures, especially in robot caregiving scenarios. While humans can often help robots recover, excessive or poorly targeted queries impose unnecessary cognitive and physical workload on the human partner. We present a human-in-the-loop failure-recovery framework for modular robotic policies, where a policy is composed of distinct modules such as perception, planning, and control, any of which may fail and often require different forms of human feedback. Our framework integrates calibrated estimates of module-level uncertainty with models of human intervention cost to decide which module to query and when to query the human. It separates these two decisions: a module selector identifies the module most likely responsible for failure, and a querying algorithm determines whether to solicit human input or act autonomously. We evaluate several module-selection strategies and querying algorithms in controlled synthetic experiments, revealing trade-offs between recovery efficiency, robustness to system and user variables, and user workload. Finally, we deploy the framework on a robot-assisted bite acquisition system and demonstrate, in studies involving individuals with both emulated and real mobility limitations, that it improves recovery success while reducing the workload imposed on users. Our results highlight how explicitly reasoning about both robot uncertainty and human effort can enable more efficient and user-centered failure recovery in collaborative robots.

Methodology

Methodology diagram

BibTeX

@inproceedings{banerjee2026modularhil,
      author    = {Banerjee, Rohan and Palempalli, Krishna and Yang, Bohan and Fang, Jiaying and Abdullah, Alif and Silver, Tom and Dean, Sarah and Bhattacharjee, Tapomayukh},
      title     = {A Human-in-the-Loop Confidence-Aware Failure Recovery Framework for Modular Robot Policies},
      booktitle = {Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI)},
      year      = {2026},
}