Learning dexterous robotic manipulation directly from human videos is fundamentally challenged by the kinematic embodiment gap and the lack of contact information that is unobservable in videos. To address these limitations, we present UniDex-ViTac, a unified visuo-tactile imitation learning framework that distills physically feasible, contact-rich trajectories generated by residual RL specialists into a single multi-task generalist policy. Crucially, our generalist operates on an expressive visuo-tactile representation that explicitly fuses global 3D point clouds with local binary tactile feedback. By effectively reasoning over both spatial geometry and local contact events, UniDex-ViTac achieves a 68.3% success rate in simulation and demonstrates robust Sim2Real transfer on a physical 16-DoF hand, achieving a 66.4% average success rate across diverse seen and unseen objects.
Our pipeline reconstructs HOI trajectories from human videos, trains per-object Residual RL specialists to produce physically feasible robot motions, then distills 10,000 successful rollouts into a single multi-task Visuo-Tactile Generalist policy.
Each point in the 512-point cloud carries a one-hot label: (1,0,0) camera points, (0,1,0) non-contacting fingertips, (0,0,1) contacting fingertips. This is combined with a 42-D proprioceptive state and a 4-D binary contact signal, enabling the policy to jointly reason over scene geometry and local contact events.
Built on ACT with a CVAE encoder, the policy Transformer takes four tokens — point cloud (PointNet++), proprioception, binary contact, and a style latent — and predicts a 30-step action chunk of wrist pose and fingertip targets, executed via DLS arm IK and analytical hand IK.
(a) Our visuo-tactile policy achieves 68.3% average success on 10 YCB objects, outperforming ACT (53.4%), Diffusion Policy (45.2%), and BC-Transformer (20.2%). (b) Point cloud and binary contact are complementary — fusing both outperforms either alone (PCD Only: 55.5%, Contact Only: 58.4%). (c) Our dual fusion (onehot label + contact token) consistently performs best across all objects.
Deployed on a Franka Emika Panda with a 16-DoF dexterous hand, our PC + Tactile policy achieves 66.4% overall success across seen and unseen objects, outperforming the PC Only baseline (54.5%).
| Category | Object | PC Only | PC + Tac |
|---|---|---|---|
| Seen | bleach_cleanser | 10/10 | 9/10 |
mustard_bottle | 7/10 | 7/10 | |
power_drill | 7/10 | 8/10 | |
potted_meat_can | 4/10 | 7/10 | |
tomato_soup_can | 3/10 | 5/10 | |
pudding_box | 2/10 | 6/10 | |
| Average | 55.0% | 70.0% | |
| Unseen | maxwell_coffee_can |
9/10 | 9/10 |
brown_salt_box | 2/10 | 4/10 | |
plastic_wine_cup | 7/10 | 6/10 | |
blue_mug | 0/10 | 2/10 | |
pringles_bottle | 9/10 | 10/10 | |
| Average | 54.0% | 62.0% | |
| Overall Average | 54.5% | 66.4% | |
Table 1: Real-world success rates (out of 10 trials) comparing the Point Cloud Only policy and the Point Cloud + Binary Tactile policy.
@inproceedings{unidex-vitac2026,
title = {UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data},
author = {Hyesung Lee and Si-Hwan Heo and Sungwook Yang},
booktitle = {ICRA 2026 Workshop: Beyond Teleoperation},
year = {2026}
}