Robust robot manipulation in unstructured environments often requires understanding object properties that extend beyond geometry, such as material or compliance—properties that can be challenging to infer using vision alone. Multimodal haptic sensing provides a promising avenue for inferring such properties, yet progress has been constrained by the lack of large, diverse, and realistic haptic datasets. In this work, we introduce the CLAMP device, a low-cost (<$200) sensorized reacher-grabber designed to collect large-scale, in-the-wild multimodal haptic data from non-expert users in everyday settings. We deployed 16 CLAMP devices to 41 participants, resulting in the CLAMP dataset, the largest open-source multimodal haptic dataset to date, comprising 12.3 million datapoints across 5357 household objects. Using this dataset, we train a haptic encoder that can infer material and compliance object properties from multimodal haptic data. We leverage this encoder to create the CLAMP model, a visuo-haptic perception model for material recognition that generalizes to novel objects and three robot embodiments with minimal finetuning. We also demonstrate the effectiveness of our model in three real-world robot manipulation tasks: sorting recyclable and non-recyclable waste, retrieving objects from a cluttered bag, and distinguishing overripe from ripe bananas. Our results show that large-scale, in-the-wild haptic data collection can unlock new capabilities for generalizable robot manipulation.
The CLAMP device is a low-cost, sensorized reacher-grabber designed for collecting multimodal haptic data in household settings. Our device is easy to build and easy to scale.
(assembly videos are at 2x speed)
The CLAMP device includes a PiTFT screen and buttons, with a graphical user interface (GUI) that guides non-experts through the data collection process.
The CLAMP dataset contains data from 16 CLAMP devices deployed in 41 households. It is
the largest open-source multimodal haptic dataset in the robotics literature, comprising
12.3 million datapoints
across 5357 household objects.
We train the CLAMP model, a visuo-haptic model trained on the CLAMP dataset to recognize object material. The CLAMP model consists of a GPT-based pretrained encoder and a learned haptic encoder.
Our learned haptic encoder can be transferred to compliance recognition without any finetuning
Our models generalize across varying robots and end-effectors, with minimal finetuning.
We showcase our models in three real-world manipulation tasks that require material & compliance recognition.
The robot sorts objects into "Trash" or "Recycle" boxes based on a material prediction from the finetuned CLAMP model. Objects are put in the "I'm not sure" box if the prediction is unknown/uncertain*.
The robot retrieves metallic objects from a bag with unknown contents based on a material prediction from the finetuned CLAMP model.
The robot retrieves only ripe bananas from a fruit basket with ripe and overripe bananas based on a compliance prediction from the finetuned haptic encoder transferred to compliance recognition.
@misc{thakkar2025clamp,
title={CLAMP: Crowdsourcing a LArge-scale in-the-wild haptic dataset with an open-source device for Multimodal robot Perception},
author={Thakkar Pranav N. and Sinha Shubhangi and Baijal Karan and Bian Yuhan (Anjelica) and Lackey Leah and Dodson Ben and Kong Heisen and Kwon Jueun and Li Amber and Hu Yifei and Rekoutis Alexios and Silver Tom and Bhattacharjee Tapomayukh},
year={2025},
eprint={TODO},
archivePrefix={arXiv},
primaryClass={cs.RO}
}