Abstract: This paper introduces REPeat, a novel Real2Sim2Real framework aimed at improving bite acquisition in robot-assisted feeding for individuals on soft diets, a critical need highlighted by the high prevalence of dysphagia in conditions such as ALS, Parkinson's disease, Stroke, and Multiple Sclerosis. Utilizing monocular depth estimation, REPeat transforms real-world RGB images into detailed 3D models to simulate the physical dynamics of various food types, from Newtonian fluids to non-Newtonian substances and granular solids. By leveraging the Material Point Method (MPM) for accurate food physics modeling, the framework allows the exploration of pre-acquisition strategies, extending the adaptability to the rheological complexity of soft diets. The Sim2Real step employs ControlNet to generate realistic images of the simulated plates for evaluating bite-acquisition strategies. Tested across 15 diverse food plate scenarios, REPeat shows success rate improvements in bite acquisition for most plates.
@inproceedings{ha2024repeat,
author = {Ha, Nayoung, and Ye, Ruolin and Liu, Ziang and Sinha, Shubhangi and Bhattacharjee, Tapomayukh},
title = {REPeat: A Real2Sim2Real Approach for Pre-acquisition of Soft Food Items in Robot-assisted Feeding},
journal = {IROS},
year = {2024},
}