Robot-assisted Inside-mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control

Rajat Kumar Jenamani, Daniel Stabile, Ziang Liu, Abrar Anwar, Katherine Dimitropoulou, Tapomayukh Bhattacharjee

Supplementary Video


Robot-assisted feeding can greatly enhance the lives of those with mobility limitations. Modern feeding systems can pick up and position food in front of a care-recipient’s mouth for a bite. However, many with severe mobility constraints cannot lean forward and need direct inside-mouth food placement. This demands precision, especially for those with restricted mouth openings, and adeptly managing various physical interactions — incidental contacts as the utensil moves inside, impulsive contacts due to sudden muscle spasms, deliberate maneuvers by the person being fed to guide the utensil, and intentional bites.

In this paper, we propose an inside-mouth bite transfer method that addresses these challenges with two key components: a multi-view mouth perception pipeline robust to tool occlusion, and a control mechanism that employs multimodal time-series classification to discern and aptly react to different physical interactions. We demonstrate the individual efficacy of these components through two ablation studies.

In a comprehensive user study, our system successfully fed 13 care-recipients with diverse mobility challenges. Participants consis- tently emphasized the comfort and safety of our inside-mouth bite transfer approach, and gave it high technology acceptance ratings, underscoring the transformative potential of this method in real- world scenarios.

Interpolate start reference image.

We propose an inside-mouth bite transfer method that uses two key components, (i) robust multi-view mouth perception,
and (ii) physical interaction-aware control, to successfully feed care-recipients with diverse mobility limitations.