Vision-based tactile sensor

A vision-based tactile sensor, also called an optical or camera-based tactile sensor, is a category of tactile sensing that measures touch by watching it rather than by reading a strain gauge, a capacitor, or a magnetic field. A small camera sits behind a soft, deformable skin, usually a silicone elastomer with a reflective coating and sometimes a printed grid of markers, and when that skin presses against an object, the camera records how the skin bends. Computer vision software then turns the resulting image into a map of contact geometry, applied force, and, in more capable designs, shear and slip, at a spatial resolution few other tactile technologies can match. The approach has become one of the most widely used ways to give robot fingertips a sense of touch, particularly in research on learning-based robot manipulation, even though the same camera-behind-gel structure that gives it such rich output also makes it bulkier and slower than simpler tactile technologies.

In brief: picture a fingertip built like a tiny periscope. A soft rubber pad forms the outer skin, and just behind it sit a miniature camera and a ring of LEDs, all sealed inside a light-tight housing. When the pad presses against something, the camera watches the underside of the rubber flex and shift, the way a fingerprint smudges against glass. Software turns that video into a map of the object's shape and the forces acting on it, without a single strain gauge or pressure switch anywhere in the design.

How it works

Nearly every vision-based tactile sensor shares the same three parts: a soft, translucent elastomer pad that forms the contact surface, a light source (usually several LEDs), and a small camera positioned to image the underside of the pad. The inner face of the pad is typically coated with a thin reflective layer, sometimes an opaque paint mixed into the silicone itself, so it behaves like a soft, diffuse mirror. Some designs also print or embed a grid of small markers, dots or lines, on or inside the gel. Because the whole assembly sits inside an enclosed housing, the sensor works in the dark and is largely unaffected by ambient lighting ^[1][2].

Researchers group vision-based sensors into two broad families by how they turn a camera image into a tactile signal, a distinction formalized in recent taxonomies of the field ^[32][33]:

• Intensity-based, or photometric, sensing. This is the original approach, introduced by Micah Johnson and Edward Adelson at the Massachusetts Institute of Technology in a 2009 paper describing what they called a "retrographic sensor" ^[1]. Several LEDs of different colors illuminate the reflective gel from different angles. Because the local surface normal at each point of the gel determines how much light from each differently colored, differently angled LED bounces back to the camera, a single captured frame effectively encodes a normal map of the deformed surface. A calibrated lookup table converts the color recorded at each pixel into a surface-normal vector, and integrating those normals across the frame reconstructs a height map of whatever pressed into the gel, all from one image and with no moving parts ^[1][3][4].
• Marker-based, or marker-tracking, sensing. Instead of decoding color into surface normals, these designs track the two-dimensional displacement of an array of physical markers: painted dots on a photometric-style gel, or, in the case of the TacTip family, an array of pins projecting from the inner surface of a soft dome, echoing the way ridges inside a human fingertip transmit skin deformation to underlying nerve endings ^[15][16]. Tracking marker motion is a comparatively direct way to recover shear and the onset of slip, which is why several photometric designs, including later GelSlim and GelSight Mini variants, now paint markers onto an otherwise photometric gel to get both kinds of signal from a single sensor ^[6][8].

Either way, the sensor's raw output is a video stream, typically an image somewhere between 320x240 and 640x480 pixels captured at 30 to 90 frames per second, rather than a handful of numbers. That is the source of both the technology's biggest strength (very high spatial resolution) and its biggest engineering headaches (bulk, data volume, and frame-rate limits), covered in more detail below.

A related but distinct optical approach uses fiber optics instead of an imaging camera. Thin strands of glass or plastic optical fiber carry light into and out of a soft pad; deformation bends, blocks, or reflects the fibers differently, and the sensor infers contact location and force from the resulting change in transmitted light intensity rather than from a reconstructed image ^[34][35]. Because the sensing region can be built with no metal and no live electronics near the contact surface, fiber-optic tactile sensors are useful in magnetic-resonance suites, high-voltage equipment, and other settings where a camera-and-circuit-board fingertip would be a liability, but they trade away most of the spatial resolution and shape-reconstruction ability that make camera-based designs distinctive ^[34]. A handful of commercial sensors sit in between: Contactile's PapillArray, built around a grid of nine soft pillars, is described by its maker and by independent researchers as an optical tactile sensor, but public technical material does not say whether it forms a full image of the contact area the way GelSight- and TacTip-style designs do, or infers pillar displacement from simpler point-by-point light measurements ^[24][25].

Types and variants

The table below covers the best-documented lineages and variants, spanning the original academic design, its direct descendants, and independent designs that use the same camera-behind-gel principle.

Scroll sideways for more →

Two 2026 developments illustrate that the field is still moving. First, engineers at Queen Mary University of London and collaborators in Italy published a sensor that encodes pressure and strain directly as a changing structural color in the material itself, so a standard low-cost camera can read high-resolution deformation without the heavier computation photometric-stereo reconstruction normally requires ^[23]. Second, XELA Robotics, whose own sensors use magnetic rather than optical sensing (see Tradeoffs, below), added software that fuses its taxel arrays with external vision-based observation, part of a broader industry move toward combining multiple tactile modalities rather than treating vision-based sensing as a self-contained solution ^[31].

Tradeoffs and key evaluation criteria

Strengths. Because the output is literally an image, feature resolution can reach tens of micrometers per pixel, finer than most discrete taxel arrays, which is why GelSight-derived sensors are also used outside robotics for surface-texture inspection ^[3][4]. Marker tracking and photometric deformation both make shear and the earliest signs of slip directly visible in the image, which several of the papers behind these designs identify as the most useful early warning that a grasped object is about to be dropped ^[5][6]. A vision-based fingertip also reports a spatial map of the entire contact patch rather than a single reading, unlike a wrist-mounted force-torque sensor, which reports one combined six-axis measurement for the whole hand or arm. Because the raw signal is already an image, it drops naturally into the same convolutional and transformer architectures used elsewhere in computer vision and deep learning, and DIGIT's roughly 15-dollar bill of materials at volume shows that at least the open designs can be cheap enough to fit on every fingertip of a hand rather than a single sample point ^[10][11].

Weaknesses. A camera needs a minimum working distance behind the gel to see the whole contact area, or, as in OmniTact, GelSlim, and the GelSight Wedge, needs mirrors or several tiny cameras folded into a small volume to avoid that depth penalty. Either way, vision-based fingertips are typically thicker than a bare piezoresistive film, a real constraint on a humanoid hand's fingertip, where space also has to fit tendons, gearing, or drive electronics ^[5][7][18]. Standard designs run at 30 to 90 frames per second, fast enough for most quasi-static grasping but capable of missing very brief events; event-based designs like Evetac push into the kilohertz range by swapping the frame camera for an event camera, the same class of sensor chip sold by companies such as Prophesee, at the cost of losing the dense, full-frame image most photometric algorithms are built around ^[20]. Because the transduction mechanism is a camera watching mechanical deformation, a plain photometric or marker-based design cannot sense temperature, humidity, or contact too light to visibly compress the skin; Meta's Digit 360 works around this by packing separate heat and odor sensors into the same housing as the camera rather than deriving those cues from the image itself ^[12]. The elastomer pad is also a consumable: GelSight sells replacement gels for the Mini as a routine accessory, and independent comparisons put commercial vision-based sensors at a few hundred to a few thousand contact cycles before the gel needs replacing ^[9]. Finally, because each sensor is effectively a small video camera, a hand fully instrumented at every fingertip means streaming several simultaneous video feeds rather than a handful of scalar readings, a bandwidth and processing load that has pushed some designs toward on-board edge AI processing (Digit 360's built-in accelerator is one example) even though most current sensors still stream raw or lightly compressed frames to a central processor ^[12].

The table below places vision-based sensing against the other three tactile transduction principles commonly used in robot hands.

Scroll sideways for more →

Piezoresistive and piezoelectric sensors change electrical resistance or generate a small charge under strain; they are thin and easy to embed across large areas but lose accuracy on sliding, shear-dominated contact ^[32][33]. Capacitive sensors, the same underlying technology as a smartphone touchscreen, are excellent at detecting very light contact but are sensitive to electromagnetic noise and static. Magnetic sensors embed a small magnet in a soft pad next to a Hall-effect chip and infer touch from how the magnetic field shifts as the magnet moves; because the magnet can move in more than one direction, magnetic designs are naturally good at multi-axis force and shear sensing and can sample far faster than a camera, but they need enough standoff to fit the magnet and can be confused by nearby ferromagnetic tools or fixtures. Paxini's dexterous hand hardware and XELA Robotics's uSkin line are both built around this approach rather than a camera, and XELA has published magnetic-interference compensation specifically to handle the metal-handling case ^[30][31]. A separate, non-optical approach worth noting for contrast is the fluid-and-electrode design used in SynTouch's BioTac fingertip, which infers force, vibration, and thermal flux from a deformable, liquid-filled skin rather than from any of the four principles above.

Use in humanoid robots

On a humanoid robot, the hand is usually the main end effector, the part of the machine that actually touches the world, and vision-based tactile sensing is one of several competing approaches to giving humanoid robot hands a sense of touch. Fingertips are by far its most common placement: a camera-behind-gel package is easiest to fit where a hand already has room for a fingertip cap, unlike full-body electronic skin coverage, which tends to favor thinner modalities such as piezoresistive film or magnetic taxels ^[30][31].

Several companies building humanoid or dexterous hands have adopted the approach directly. Sharpa's SharpaWave hand pairs a fingertip matrix of more than a thousand tactile pixels with an embedded micro-camera, which the company describes as letting the hand see and feel what it touches, and reports detecting forces as small as 0.005 newtons at up to 180 frames per second ^[26][27]. Sharpa was founded at the end of 2024 by three co-founders of Hesai, the automotive lidar maker, and is based in Singapore ^[27]. Daimon Robotics, a company founded in 2023 and based in Shenzhen and Hong Kong that grew out of research at the Hong Kong University of Science and Technology, has built a millimeter-thin vision-based tactile sensor that it embeds directly into the fingertips of its DM-Hand1 dexterous hand, and demonstrated the sensor at the ICRA 2025 robotics conference ^[28][29]. On the open research side, Meta's Digit Plexus circuit board is designed to connect several DIGIT- or Digit 360-class sensors across a single hand and is built to integrate with Wonik Robotics's next generation of the Allegro Hand ^[12][13][14].

Several other widely covered humanoid-hand programs have described adding dedicated tactile sensing to their fingertips without publicly specifying which physical principle the sensors use, so it would not be accurate to count them as confirmed vision-based tactile adopters based on public information alone. That gap between "has tactile sensors" and "has disclosed which kind" is common across the humanoid industry, since fingertip sensor choice is often treated as proprietary.

Because the raw signal from a vision-based sensor is an image, it fits naturally into the same imitation learning and reinforcement learning pipelines already used for camera-based robot control, rather than requiring a separate signal-processing stack. Meta's open-source TACTO simulator was built specifically to generate synthetic GelSight- and DIGIT-style tactile images so that policies can be pretrained in simulation before ever touching a real sensor, a tactile-specific instance of the broader practice of sim-to-real transfer. Meta's Sparsh models take this further: pretrained on more than 460,000 tactile images using self-supervised learning, Sparsh is presented as a general-purpose touch encoder, sometimes described as a tactile foundation model, that the company reports improves substantially over sensor-specific models when labeled training data is scarce ^[12]. Researchers have also begun exploring tactile images as an additional input alongside cameras for vision-language-action model policies, though, as with vision-only foundation models a few years earlier, this remains an active research direction rather than a settled part of production humanoid software stacks.

Suppliers and landscape

Vision-based tactile sensing spans a wider range of makers than most other tactile modalities, largely because the core design, a small camera plus a soft gel, has been open-sourced by multiple university labs rather than locked behind a single company's patents. The table below covers the most established academic origins and commercial makers of vision-based tactile hardware specifically; it excludes companies such as Paxini and XELA Robotics, whose flagship products use magnetic rather than optical transduction (see Tradeoffs, above).

Scroll sideways for more →

GelSight Inc. occupies an unusual position in this landscape: founded in 2011 by Janos Rohaly, Bill Yost, and Micah "Kimo" Johnson (the same Johnson who co-authored the original 2009 paper) as a spinout of MIT's Adelson lab, it now both sells its own commercial sensors and manufactures DIGIT and Digit 360 under contract for Meta, making it simultaneously a competitor to and a supplier for the open-source side of the field ^[2][10][12].

See also

• Tactile sensing
• Dexterous hand
• Electronic skin
• Force-torque sensor
• Robot manipulation
• Humanoid robot hands
• Computer vision
• End effector
• Paxini Technology

References

1. Johnson, M. K. and Adelson, E. H. "Retrographic sensing for the measurement of surface texture and shape." IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. ↩
2. GelSight, Inc. "About GelSight." gelsight.com/about/. ↩
3. GelSight, Inc. "Tactile sensing technology." gelsight.com/tactile-sensing-technology/. ↩
4. "GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force." Sensors (MDPI), vol. 17, no. 12, 2017. ↩
5. Donlon, E., Dong, S., Liu, M., Li, J., Adelson, E., and Rodriguez, A. "GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger." IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018. arXiv:1803.00628. ↩
6. Taylor, I. H., Dong, S., and Rodriguez, A. "GelSlim 3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Finger." IEEE International Conference on Robotics and Automation (ICRA), 2022. arXiv:2103.12269. ↩
7. Wang, S., She, Y., Romero, B., and Adelson, E. "GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger." IEEE International Conference on Robotics and Automation (ICRA), 2021. ↩
8. GelSight, Inc. "GelSight Mini." gelsight.com/gelsightmini/. ↩
9. "Introducing the GelSight Mini: A Human Resolution Tactile Sensor for Engineers and Scientists." Business Wire, September 21, 2022. ↩
10. Lambeta, M., Chou, P., Tian, S., Yang, B., Maloon, B., Most, V. R., Stroud, D., Santos, R., Byagowi, A., Kammerer, G., Jayaraman, D., and Calandra, R. "DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation." IEEE Robotics and Automation Letters, vol. 5, no. 3, 2020. arXiv:2005.14679. ↩
11. Ackerman, E. "Why Facebook (Or Meta) Is Making Tactile Sensors for Robots." IEEE Spectrum, 2020. ↩
12. Meta AI (FAIR). "Advancing embodied AI through progress in touch perception, dexterity, and human-robot interaction." ai.meta.com/blog/fair-robotics-open-source/, 2024. ↩
13. Heater, B. "Meta is making a robot hand that can 'feel' touch." TechCrunch, October 31, 2024. ↩
14. "GelSight, Meta AI release Digit 360 tactile sensor for robotic fingers." The Robot Report, 2024. ↩
15. Chorley, C., Melhuish, C., Pipe, T., and Rossiter, J. "Development of a tactile sensor based on biologically inspired edge encoding." International Conference on Advanced Robotics (ICAR), 2009. ↩
16. Ward-Cherrier, B. et al. "The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies." Soft Robotics, vol. 5, no. 2, 2018. ↩
17. Lin, C., Zhang, H., Xu, J., Wu, L., and Xu, H. "9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation." IEEE Robotics and Automation Letters / ICRA, 2024. arXiv:2308.14277. ↩
18. Padmanabha, A., Ebert, F., Tian, S., Calandra, R., Finn, C., and Levine, S. "OmniTact: A Multi-Directional High-Resolution Touch Sensor." IEEE International Conference on Robotics and Automation (ICRA), 2020. arXiv:2003.06965. ↩
19. Zhang, L. et al. "Tac3D: A Novel Vision-based Tactile Sensor for Measuring Forces Distribution and Estimating Friction Coefficient Distribution." arXiv:2202.06211, 2022. ↩
20. Funk, N., Helmut, E., Chalvatzaki, G., Calandra, R., and Peters, J. "Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation." arXiv:2312.01236, 2023. ↩
21. Toyota Research Institute. "Sensing is Believing: More Capable Robot Hands with the Soft Bubble Gripper." tri.global/news, 2020. ↩
22. "Toyota Research Institute Shares Design to Help Advance the Field of Soft Robotics." PR Newswire, May 2021. ↩
23. "Robots can now 'see' touch thanks to a new color-changing tactile sensor." Tech Xplore, July 2026, reporting on Sasso, G. et al., Science Advances, July 3, 2026 (DOI: 10.1126/sciadv.aee5236). ↩
24. Contactile. "Products." contactile.com/products/. ↩
25. "Robust Learning-Based Incipient Slip Detection using the PapillArray Optical Tactile Sensor for Improved Robotic Gripping." arXiv:2307.04011, 2023. ↩
26. "World's most advanced robotic hand pairs vision and touch sensing." Interesting Engineering, 2026. ↩
27. "Singapore-based Sharpa surfaces as robotics venture linked to Hesai founders." KrASIA. ↩
28. "Daimon Robotics Secures Over RMB 100 Million in Angel++ Round Financing, Ushering in a New Era of Tactile Dexterity in Robotics." GlobeNewswire, August 15, 2025. ↩
29. "DAIMON Robotics Wants to Give Robot Hands a Sense of Touch." IEEE Spectrum. ↩
30. "Robot Sensor Startup PaXini Raises $145 Million, Tops $1.4 Billion Valuation." Caixin Global, March 7, 2026. ↩
31. "XELA Robotics Upgrades uSkin Sensors with Metal Handling, High-Speed Communication." The Robot Report / New Equipment Digest. ↩
32. "Classification of Vision-Based Tactile Sensors: A Review." arXiv:2509.02478, 2025. ↩
33. "Hardware Technology of Vision-Based Tactile Sensor: A Review." IEEE, 2022. ↩
34. "Color-Coded Fiber-Optic Tactile Sensor for an Elastomeric Robot Skin." arXiv:1908.03687, 2019. ↩
35. "Integration of Fiber-Optic Sensor Arrays into a Multi-Modal Tactile Sensor Processing System for Robotic End-Effectors." PMC, National Institutes of Health. ↩