TACTIC

Overview

Whole-arm contact is fundamental to human manipulation: people use the forearm and upper arm to brace, hold, guide, and reposition objects and bodies, distributing forces over a larger area. Robots need the same capability in physical caregiving, general physical human–robot interaction, and manipulation of large, unwieldy, or fragile objects in clutter.

TACTIC is a receding-horizon controller for whole-arm manipulation: it fuses vision and distributed tactile sensing into a contact-centric state and plans with a hybrid predictive model, trading task progress against whole-arm force regulation.

Why is this hard?

Purely learned predictors can produce physically inconsistent contact and force estimates under distribution shift; purely analytical models miss the multimodal, partially observed interactions. TACTIC couples the two.

Method

TACTIC uses receding-horizon, sampling-based MPC (MPPI): receding-horizon control enables continual replanning as contacts form, break, and shift across the arm, and sampling-based optimization can evaluate diverse candidate actions under nonconvex, contact-dependent dynamics. At each step, TACTIC encodes the latest observations, samples action sequences shaped by the active contact Jacobians, rolls each through a hybrid predictive model, scores the rollouts, executes the best first action, and replans.

A · Contact-centric representation

Tactile readings are spatially sparse, available only upon contact, and localize contact on the arm rather than in the scene; many tasks require dense, anticipatory reasoning about where in the scene contact is occurring or likely to occur next. The proximity mask is an image-aligned 2D representation of robot-object proximity that grounds impending and active contact in the scene frame. The RGB image is segmented into robot and object masks with a finetuned YOLOv11, pixels are unprojected into 3D point sets using the camera intrinsics, and each robot point's distance to the closest object point (kd-tree nearest-neighbor) maps to pixel intensity: near is bright, far is dark. Each taxel (tactile pixel) with force above a threshold is then projected onto the image via forward kinematics and rendered as a unit-valued blob; the final mask is the per-pixel maximum of the two.

RGB, depth, and the proximity mask share a DINOv3 backbone in a batched forward pass, with depth and mask replicated to three channels. Taxel forces, a vector of normal forces under a fixed ordering along the arm, are encoded by a convolutional network with 1D CoordConv layers whose taxel-index channel lets it learn location-dependent contact features. Proprioception passes through an MLP, with joint angles in a sin-cos representation to avoid wrap-around. The dynamics model conditions on a history of the resulting latents.

B · Contact-aware action sampling

The set of active contacts can change from step to step. If contact enters only through the scalar rollout cost, the sampler adapts slowly: covariance updates lag the current contact interaction, especially with a small particle budget. TACTIC instead shapes perturbations at every MPC step for minimal-lag, contact-conditioned exploration toward the contact-normal subspace.

Each measured contact defines a contact Jacobian whose normal row maps joint velocities to normal velocity at that contact; stacking these rows spans the joint directions that can change normal motion at the current contacts. Each contact is weighted by its measured force (with saturation) so a light brush or noisy taxel does not dominate sampling. A ridge-regularized filter built from this weighted Jacobian amplifies the component of each perturbation that changes normal contact velocity relative to the component that leaves it unchanged. Finally, sampled actions are projected onto a safety cone so they never command motion further into a contact already above the safety force threshold.

C · Contact-centric hybrid predictive model

Small changes in arm configuration can shift where contact occurs and whether forces increase or decrease. The hybrid predictive model separates what can be computed reliably (robot kinematics) from what is difficult to model analytically (contact formation and interaction dynamics). The analytical branch rolls out deterministic kinematics per candidate action sequence, yielding joint states, task-space poses, and contact Jacobians along the predicted configurations; it rejects infeasible motion and tracks how actions move active contacts along their normals. The learned branch follows the JEPA recipe for robotic world models: a ViT predictor with a frame-causal attention mask, action-conditioned through adaptive layer normalization, rolls the latent state forward, and lightweight decoders map predicted latents to joint positions and per-taxel binary contact. Training is offline, with a teacher-forced latent consistency loss plus a multistep rollout loss that reduces drift when the model consumes its own predictions.

Simulation Experiments

We first evaluate in simulation to isolate each design choice. Q1: does the contact-centric representation improve task success and safety? Q2: does contact-aware action sampling improve interaction response, safety, and smoothness? Q3: does hybrid modeling improve success and efficiency versus kinematics or learned dynamics alone? Tables report success rate (SR), time to completion in seconds (TTC), and force violations (FV, count of unsafe-force timesteps); arrows mark the better direction.

Environments, data, and training

Data for Maze, Bathing, and Reach comes from RRT-generated feasible start/goal configurations and random-play trajectories; Granular uses the DINO-WM dataset. All learned components are trained offline; no method uses online interaction. All ablations run in Maze to minimize confounds.

Q1 · Contact-centric sensing improves safety and performance

We ablate tactile sensing and the proximity mask in Maze. Removing both drops success from 87% to 64% and more than doubles unsafe-force timesteps. The vision-based mask alone recovers part of the gap, indicating it provides anticipatory information about likely contact regions beyond sparse tactile events.

Q2 · Contact-aware sampling reduces unsafe interaction

Against a variant using the base sampler with the same model and costs, injecting contact geometry into the sampler yields roughly two-thirds fewer force violations and half the mean jerk.

Q3 · Hybrid rollouts beat either model alone

Kinematics-only rollouts provide geometric grounding but cannot anticipate contact state changes; latent-only rollouts capture interaction patterns but can drift without analytical structure. Hybrid rollouts achieve the best success/safety trade-off.

Against model-based baselines

We compare against two offline model-based RL baselines trained on the same data: DreamerV3, which learns a latent dynamics model with actor and value functions (an end-to-end way to act from a learned model, without contact-aware sampling or analytical rollouts), and TD-MPC2, a decoder-free method that learns latent dynamics and value estimates and plans in latent space. TACTIC achieves the highest success and lowest force violations on Maze and improves both on Bed Bathing.

Hybrid modeling also helps pretrained world models: adding a kinematics-based goal cost to the planner, with the pretrained encoder and predictor kept intact, lets V-JEPA 2 reach comparable performance with ~34% less finetuning data on Reach and reduces frozen DINO-WM's rollout Chamfer distance by ~26% on Granular.

Real-World Experiments

Real-world deployment requires real-time control under sensing noise, actuation delays, and changing contact conditions. We evaluate on (i) a 3D maze requiring highly constrained multi-contact trajectories, and (ii) contact-intensive tasks on a life-size manikin (~46 lb, vinyl plastic, of the type used for nurse training): rolling it from its back onto its side, and lifting and repositioning its legs to a target configuration.

Baselines and results

The model-based RL baselines (DreamerV3, TD-MPC2) underperformed in simulation, so we compare against Diffusion Policy, an imitation baseline trainable from limited expert demonstrations, and pair each autonomous trial with an expert teleoperation episode under the same start/goal configuration as a reference.

TACTIC achieves higher success than Diffusion Policy on two of three tasks and is comparable on Maze. On every task it produces fewer force-threshold violations than the matched expert teleoperation episodes: contact-aware planning regulates forces beyond the demonstration distribution, i.e., better than the data it was trained on.

Diffusion Policy shows no maze violations because it usually enters from the top of the maze and fits into the final configuration without going through the maze.

Open Hardware

Whole-arm manipulation data is scarce partly because the hardware to collect it is scarce. Alongside the paper we release both pieces of ours: design files, assembly instructions, and code.

BibTeX

@inproceedings{madan2026tactic,
  title     = {{TACTIC}: Tactile and Vision Conditioned Contact-Centric
               Control for Whole-Arm Manipulation},
  author    = {Madan, Rishabh and Xie, Angchen and Saak, Samantha and
               Blanco, Andres and Lee, Dohyeok and Brown, Sarah Grace and
               Yan, Yunting and Zolotas, Mark and Barreiros, Jose and
               Bhattacharjee, Tapomayukh},
  booktitle = {Robotics: Science and Systems (RSS)},
  year      = {2026}
}