T-Rex: Tactile-Reactive Dexterous Manipulation

T-Rex: Tactile-Reactive Dexterous Manipulation is a 2026 robot learning research system for contact-rich, bimanual manipulation. A 34-author team from institutions including the University of California, Berkeley, NVIDIA, Stanford University, Panasonic, La Sapienza University, and ItalAI introduced it in an arXiv preprint submitted on June 15, 2026 and revised on June 18.^[1] T-Rex combines a tactile-synchronized robot dataset, a variable-rate vision-language-action policy, and a three-stage training recipe. Its slow pathway plans from images and language, while a faster pathway uses fingertip force and deformation signals to revise actions during contact.^[1][2]

The project released code, two checkpoints, and part of its robot dataset. The paper describes a 100-hour collected corpus containing 7,755 episodes, but the current public subset is about 50 hours and 5,464 episodes.^[3][4] Those are different scopes. The system's reported task results are author-run evaluations from a preprint. No independent physical replication was located by Aug. 22, 2026.^[1][12][13]

Key facts

Scroll sideways for more →

Hardware and data

The physical setup is a fixed-base bimanual Vega-1. Each arm has seven actuated joints, and each Sharpa Wave hand has 22 degrees of freedom. The phrase "22 DoF" therefore describes each hand, not the entire robot. The wheels, torso, and head were fixed during data collection. A head camera and two wrist cameras provided RGB observations, while five tactile sensors on each hand measured contact at all 10 fingertips.^[1][4]

The current public midtraining data stores observation.state and action as 58-dimensional joint-position vectors. Their layout is seven left-arm joints, 22 left-hand joints, seven right-arm joints, and 22 right-hand joints. A separate 60-dimensional tactile-force field stores one 6-axis force and torque wrench for each fingertip. The release also includes three 640 x 360 RGB streams, 10 raw tactile-image streams, and 10 deformation-map streams. The high-level collection loop records at 30 Hz while a lower-level controller runs at 300 Hz.^[4]

That public 58-dimensional schema differs from the downstream posttraining and evaluation representation. The repository calls the latter eef-62: a 62-dimensional end-effector-plus-hand representation with relative end-effector control for the two arms and absolute joint control for the fingers. Its training data contains 16-step action chunks. Treating 58 and 62 as interchangeable would mix a public joint-target dataset with the task-specific policy interface.^[1][3]

Collected corpus and public subset

The full corpus described in the paper contains 7,755 episodes totaling 100 hours. It covers 207 household objects, 22 motor primitives, and 502 object-primitive combinations retained after manually removing physically infeasible pairings. The authors report a median episode length of 29.8 seconds, an interquartile range of 21.0 to 41.1 seconds, and collection over 10 weeks. The 22 motor primitives are behavior labels and should not be confused with the 22 degrees of freedom in each hand.^[1]

The current Hugging Face dataset is a subset. Its card lists 5,464 episodes, 5,473,459 frames, about 50 hours at 30 frames per second, 5,370 human-verified language instructions, 207 objects, and 22 motor primitives. It is stored in LeRobotDataset v3.0 and was about 1.53 TB at the cutoff.^[4] The code repository and Jim Fan's later launch post likewise describe about 50 public hours, even though the paper and parts of the project page call the complete 100-hour research corpus open source.^[3][5]

The paper says a commercial vision-language model drafted each instruction from four to six sampled head-camera frames plus the object and motor-primitive labels. Human annotators then checked the text. It also says teleoperators consented, third-party people do not appear in the released RGB streams, and reset frames containing operators' hands were clipped.^[1] These statements describe the authors' collection and review process, not an independent privacy audit.

Variable-rate tactile policy

T-Rex is a vision-language-action model organized as a Mixture-of-Transformer-Experts. Here the term refers to three specialized transformer pathways rather than sparse token routing among interchangeable experts.^[1]

Scroll sideways for more →

Action generation uses conditional flow matching. For the reported inference schedule, the action expert performs six Euler steps from noise to an intermediate point at tau = 0.4. The tactile expert completes four more steps from that point to the final action. Within a 16-action chunk, tactile updates occur at offsets 0, 4, 8, and 12. Because the tactile expert reuses cached vision-language key-value state, it can update without rerunning the full visual stack.^[1]

The repository summarizes the two operating rates as about 5 Hz for slow action denoising and about 20 Hz for tactile refinement.^[3] These are release measurements for the authors' stack, not universal guarantees for other robots or accelerators. "Asynchronous" describes the different update schedule. The disclosed server uses an execution lock to serialize access while the slow pathway refreshes the shared cache.^[1][3]

Tactile information is encoded in several forms. A per-finger VQ-VAE processes a 16-frame force and torque history into temporal tokens. The model also projects the current force vector directly and encodes the current deformation map with a convolutional network. The paper reports a 256-dimensional VQ-VAE embedding and a 64-entry codebook. Combining temporal force, instantaneous force, and spatial deformation is intended to distinguish changes such as slip, impact, and sustained contact that a single static tactile frame may miss.^[1]

Training

The authors use three stages. First, the latent and action experts are pretrained on 22,889 hours of egocentric human video without tactile input. Second, robot midtraining uses the 100-hour collected tactile corpus to align those visual priors with executable bimanual actions and train the tactile expert. Third, skill-specific posttraining uses about 100 demonstrations for each evaluation task.^[1]

The paper's supervised fine-tuning configuration reports 24 H100 GPUs in bf16. The public pretraining checkpoint is described as action and latent expert alignment on about 22,000 tactile-free episodes. The midtraining checkpoint adds cascaded tactile flow and an embedded VQ-VAE.^[3][6][7] Both are derived from the Qwen3-VL-2B-Instruct base model.^[3][8]

The repository's main branch contains posttraining and inference code. Pretraining and midtraining scripts are in a separate full-pipeline branch. The corresponding pretraining and complete midtraining corpora are not included in the code repository, so a user starts from the released checkpoints and supplies task-specific demonstrations.^[3]

Author-run evaluation

The authors tested T-Rex and six baselines on 12 tabletop tasks, including page turning, egg transfer, toothpaste application, separating nested cups, opening a lock, extracting a card, and screwing in a lightbulb. Each method received 16 trials per task with randomized object positions and rotations. Multi-stage tasks used additive, progress-based rubrics, so the reported "success rate" can award partial completion rather than only a binary completed or failed outcome.^[1]

Scroll sideways for more →

The exact leading comparison is 65 percent versus 35 percent, an absolute difference of 30 percentage points.^[1][2] The percentage-point unit matters because a relative change would use a different denominator. The project page reports T-Rex as the highest-scoring method on all 12 tasks, but these remain the authors' own trials on their hardware and rubric.^[2]

The authors reproduced or adapted the baselines rather than importing results from the original teams. ViTacFormer was changed from a 21-DoF hand setup to the 22-DoF Sharpa hardware, for example, and Tactile-VLA was adapted from GelSight input to 6-axis wrench input. All methods used the T-Rex team's robot, action space, posttraining data, and protocol.^[1] This improves within-study consistency, but it does not constitute an external benchmark submission.

On a six-task ablation, the authors report 65 percent for the full model, 42 percent without tactile input, and 60 percent without asynchronous refinement. Those are differences of 23 and 5 percentage points within the ablation, respectively.^[1] They support the paper's design argument under its test conditions, but do not isolate how the same components would behave on another hand, sensor suite, or task distribution.

The main table reports no confidence intervals. The paper is an arXiv preprint, and no independent physical replication was found by the cutoff. An independent robotics researcher published a detailed technical summary, but did not rerun the experiments.^[12] MANUS, the glove supplier used in data collection, published a case study that explicitly says all results and metrics came from the paper's authors.^[13]

Released artifacts and licenses

The artifacts do not share one blanket license.

Scroll sideways for more →

The checkpoint tags do not remove terms that apply to upstream software, weights, hardware drivers, or other dependencies. The empty checkpoint cards also omit intended-use, safety, evaluation, and training-data sections commonly needed to assess a physical-control model.^[6][7]

Reproducibility, security, and safety limits

The release supports inspection and task-specific fine-tuning, but it does not reproduce the paper end to end from public inputs. Only about half of the collected 100-hour corpus is currently public, and the human-video pretraining corpus is not included. The hardware pipeline also depends on a specific dual-arm robot, tactile hands, MANUS gloves, VIVE trackers, calibration, manufacturer control software, and a manufacturer inverse-kinematics component.^{[1][3][4][13]}

Two repository details can affect repeatability. First, pyproject.toml and requirements.txt disagree on several versions, including Transformers 4.57.3 versus 4.53.1, Tokenizers 0.22.2 versus 0.21.2, and Hugging Face Hub 0.36.2 versus 0.33.2.^[10][11] Second, the checkpoint loader uses strict=False and filters shape mismatches. The repository says mismatched layers fall back to initialization values and usually need resumed training.^[3] A run can therefore load without being byte-for-byte equivalent to the authors' configuration.

The inference package exposes a single ZMQ request-reply server and a robot-side client. The released server binds to tcp://0.0.0.0 on its configured port and passes bytes received from the socket directly to pickle.loads; no authentication or transport encryption appears in that request path.^[14] Python's official documentation warns that malicious pickle data can execute arbitrary code when it is unpickled and says only trusted data should be accepted.^[15] This is a concrete reason not to expose the research server to an untrusted network. It is not evidence that a deployment was compromised, but it means the release should be understood as laboratory integration code rather than a hardened control service.

The paper does not provide a certified safety case, force-limit validation, adversarial command evaluation, or cross-platform robustness study. Its own failure analysis reports a lightbulb collision, a dropped key, imprecise egg placement, unintended multi-finger contact, excessive toothpaste force, and card-extraction misalignment. The authors also identify tactile-sensor distortion, calibration drift, cross-device variation, missing dense palm sensing, and long-horizon tasks with tight tolerances as open problems.^[1] These boundaries prevent the 12-task result from establishing reliable general-purpose dexterity or safe autonomous operation outside the evaluated setup.

References

1. ^Dantong Niu et al. "T-Rex: Tactile-Reactive Dexterous Manipulation." arXiv:2606.17055v2, June 18, 2026. arxiv.org/...2606.17055v2
2. ^T-Rex research team. "T-Rex: Tactile-Reactive Dexterous Manipulation." Project website. Accessed Aug. 22, 2026. tactile-reactive-dexterous.github.io
3. ^Zhuoyang Liu et al. "T-Rex: Official repository of T-Rex: Tactile-Reactive Dexterous Manipulation." GitHub repository. Accessed Aug. 22, 2026. github.com/...T-Rex
4. ^Zekai Wang et al. "T-Rex Dataset." Hugging Face dataset repository. Accessed Aug. 22, 2026. huggingface.co/...trex_dataset
5. ^Jim Fan. "The sense of touch is the most criminally under-explored modality in robotics." X, Aug. 21, 2026. x.com/...2090832821036470626
6. ^miniFranka. "T-Rex_pretrain_mecka22k_epoch1." Hugging Face model repository. Accessed Aug. 22, 2026. huggingface.co/...T-Rex_pretrain_mecka22k_epoch1
7. ^miniFranka. "T-Rex_midtrain_mecka23k_ucb100_vqvae_epoch6." Hugging Face model repository. Accessed Aug. 22, 2026. huggingface.co/...ain_mecka23k_ucb100_vqvae_epoch6
8. ^Qwen Team. "Qwen3-VL-2B-Instruct." Hugging Face model repository. Accessed Aug. 22, 2026. huggingface.co/...Qwen3-VL-2B-Instruct
9. ^Regents of the University of California. "T-Rex software license." GitHub, 2026. github.com/...LICENSE
10. ^Zhuoyang Liu et al. "T-Rex pyproject.toml." GitHub. Accessed Aug. 22, 2026. github.com/...pyproject.toml
11. ^Zhuoyang Liu et al. "T-Rex requirements.txt." GitHub. Accessed Aug. 22, 2026. github.com/...requirements.txt
12. ^Lixin Xu. "Paper Notes: T-Rex: Tactile-Reactive Dexterous Manipulation." June 19, 2026. davidlxu.github.io/...t-rex-paper-notes
13. ^MANUS. "T-Rex: Advancing Tactile-Reactive Dexterous Manipulation with MANUS Gloves." June 18, 2026. manus-meta.com/...s-manipulation-with-manus-gloves
14. ^Zhuoyang Liu et al. "Real-world ZeroMQ inference server." T-Rex `scripts/test.py`, GitHub. Accessed Aug. 22, 2026. github.com/...test.py
15. ^Python Software Foundation. "pickle: Python object serialization." Python 3 documentation. Accessed Aug. 22, 2026. docs.python.org/...pickle