When Does a Robot Policy Need Tactile Input?

Model and product details here are current to August 2026.

I first walked into an automotive plant in 2006 and have kept returning for MES work. Much of that work meant deciding what instruction could be handed to a machine and what still needed a person’s judgment.

Connectors were still mated by hand, harnesses routed by feel, and fasteners started by someone who could distinguish a good thread from a cross-thread. I could put a torque setpoint in a message. The person at the station also knew what it should feel like when the part seated.

When people explain why those jobs remain manual, they often say robots cannot feel. The prediction is that these jobs will be automated once they can.

I went looking for the evidence behind that conclusion, expecting to find it. Mostly I did not. Tactile sensing helps a great deal on a narrow set of tasks, does very little on many others, and in one careful study a policy given live touch performed worse than one without it. That pattern changed what I would tell someone deciding whether to add tactile input to a learned policy.

I am using touch narrowly here: tactile sensing at the contact surface, fed into the policy. A wrist force-torque sensor, motor current or a contact switch read by a conventional controller is a different design choice. Several tasks below may need one of those without needing tactile tokens in a vision-language-action model.

The boundary was already visible in automotive plants

The contrast I keep returning to is between a fixed welding cell and final assembly. A fixture can make the location of a weld predictable. A cable changes shape while someone handles it, and a gripper may hide the point where a connector seats. Automotive connectors are designed to click when they seat. That confirmation can reach the person fitting the part even when the contact is out of view.

I would not treat that contrast as proof that missing touch caused the automation boundary. Product variety, fixtures, quality requirements and economics also matter. It gives me a more specific question to take to the studies: which state must the cell observe, and what could make it observable?

Where touch helps

There are states a camera often cannot recover, and the reason is structural rather than a matter of resolution. Slip, contact force, whether a part has seated: at the moment those states matter, the gripper may be covering the contact. Wrist cameras can reduce that occlusion, but they do not always remove it.

The current revision of Tactile-WAM reports 32.7 per cent overall success on ManiFeel against 15.6 for its RGB-only DreamZero baseline. It was strongest on four contact-dominated tasks, though it was not the best system overall. In one study of dexterous insertion the vision-only policy sat at zero on a USB connector because it could not align the connector to the port. Both results come from single papers reporting against their own baselines, and I have not seen either reproduced, so treat them as indicative rather than settled. In the USB study, the camera observation did not contain the information needed to align the connector, so making the vision model larger would not supply it.

Those results are why I assumed touch was the missing ingredient.

What the controlled comparisons found

TacO trained separate policies across six tactile sensors and three tasks. That gave me a comparison across sensors and tasks instead of another result against one baseline. The paper put it this way: “the usefulness of tactile information depends strongly on sensor modality, material properties, and the specific manipulation tasks.”

On plug insertion every tactile policy beat vision-only by a wide margin. On lifting and placing a visually identical can whose mass varied, four of six tactile policies improved on vision-only, one tied and one worsened. The camera could not see the mass, but two sensor-policy pairs still failed to improve on vision-only.

Then a result that stopped me. A group pretrained a policy with a cheap tactile sensor and switched the sensor off for the task itself. Vision-only inference improved a USB-plugging task by up to 65 per cent, and the vision-only policies outperformed the ones that kept the tactile feed running. The authors attribute the reversal to bead drift in the BeadSight sensor, which created a train-test distribution shift in the live tactile stream.

Two of the studies I have now cited are about plugging in a USB connector and they point in opposite directions. In one, vision-only scored zero. In the other, vision-only won. Putting them side by side is what made me stop treating the task label as the unit of analysis: the first had tight-tolerance alignment and occluded contact, while the second had a live sensor whose observations drifted between training and evaluation, so “USB insertion” named two different information problems. I now ask two questions instead: was the decisive state visible, and did the added sensor stay trustworthy?

The studies usually report task success, but on a line I would also want to know whether the policy damaged the part. A policy may complete an insertion while applying damaging force; a seat rate does not tell you rework or part damage unless the study measured them. If the reason you want force sensing is to protect the part rather than finish the motion, define that metric separately. That is the same problem as watching conductivity on a boiler-water train and finding it did not tell us what we actually needed to know, which is why silica got its own monitor.

What the major open policies can see

Then I looked at what the shipping models take as input.

GR00T N1.7 takes images, language and robot state. π0 takes images, language and robot state. OpenVLA takes an image and an instruction. None of those major open checkpoints consumes touch while it is running.

The hardware is running ahead of that interface. NVIDIA’s announced Isaac GR00T reference humanoid design, expected in late 2026, pairs a Unitree H2 Plus body with Sharpa Wave tactile hands and Jetson Thor. N1.7’s model card still lists vision, state and language, not tactile input. Research systems such as TacVLA do feed touch into a policy, but they remain sensor- and task-specific rather than a plug-compatible path into those general checkpoints.

I first read all that as a gap waiting to be filled. I now read it as integration work the general models have not absorbed yet, which is a more useful distinction for someone specifying a cell.

How useful tactile data can still hurt a policy

Tactile datasets are small and sensor-specific compared with vision-language data, so a policy can lean on details that do not survive a new sensor, object or day. In the BeadSight study, live tactile input hurt once the beads drifted. That result is why I would test drift explicitly instead of assuming the sensor will behave the same way every day.

A different problem shows up during fine-tuning. A vision-language model already knows what a mug is and what “behind” means, none of which was paid for with robot data. If every tactile-action gradient updates that backbone, a small sensor-specific correction set can rewrite features the robot still needs before contact.

Physical Intelligence measured this with robot-action training: gradients from a newly added continuous-action expert hurt language following and knowledge transfer. Its knowledge-insulation method still adapted the backbone with discrete action and vision-language objectives, but stopped the continuous-action gradients at the backbone.

A separate TACO tactile post-training preprint applied that idea to π0.5. Despite the similar name, this is a different study from the TacO sensor comparison above. The version without knowledge insulation finished 32 percentage points below the insulated version on average across six tasks. The authors report that it learned some contact behaviour while becoming less accurate on the approach and alignment before contact. In the insulated version, the PaliGemma vision-language backbone stayed frozen while the action expert and the force and advantage encoders learned the corrections. That is one new paper, and the pre-contact diagnosis is partly qualitative, but it is enough for me to treat full-model tactile fine-tuning as a risk rather than a harmless default.

GR00T N1.5 made a related architectural choice without tactile input. NVIDIA froze its Eagle vision-language model during both pretraining and fine-tuning while a separate diffusion transformer learned robot actions from its embeddings. I see that as useful precedent for protecting the backbone, not evidence that GR00T itself accepts touch.

Touch can also interfere while the policy is running even when no weights are being rewritten. Most of an episode is not in contact. TacVLA found that feeding tactile tokens through every stage hurt both its disassembly and in-box picking tasks, so it gated those tokens on whether contact was present. That is an input-routing problem, not forgetting in the core model.

I would rerun the pre-contact reach after fine-tuning, then run it again with tactile tokens enabled and disabled. Those tests catch different failures.

Sensor designs do not transfer cleanly

TacO found very different results across sensors. I wanted to understand why.

Vision and language work across robots because large shared encoders already exist. Touch is beginning to get them too. Sparsh pretrained general-purpose representations on more than 460,000 tactile images. UniTouch learned across heterogeneous sensors. AnyTouch combines four sensors and 1.4 million multimodal sample pairs. Saying there is no shared tactile foundation would now be wrong.

They do not yet provide plug-and-play control. A GelSight or DIGIT sensor images the shading of a reflective gel as it deforms under contact. A TacTip images the displacement of an array of 3D-printed pins. Both hand you pixels, but the pixels arise from different mechanisms. TacVerse finds that transfer across sensor designs degrades unevenly: shape transfers more reliably, while force regression is sensitive to the sensor and improves with sensor-specific examples. Force regression is often the reason you fitted the sensor.

Sparsh’s 460,000-plus images and AnyTouch’s 1.4 million pairs are substantial tactile corpora. They remain orders of magnitude below the image-text side and combine different sensor mixes and supervision. A result on one gel is useful evidence, but not yet a calibration certificate for the array you are about to buy.

Using touch only during training

If touch is awkward to feed a policy at run time but informative while training, one option is to use it only during training. HapticVLA computes rewards from tactile maps that penalise excessive grip force and slip, trains a teacher on them, then distils a compact tactile token into a student that sees only vision and robot state. It reports 86.7 per cent mean success and claims to beat baselines including ones handed live tactile feedback. That is one lab with 310 real episodes, so I would want it reproduced, but it points the same way as the pretraining result above.

The teacher’s rewards encode how firmly to hold things and when a grip is drifting. The student can learn a class-level prior, such as how firmly to hold an egg, because eggs look like eggs. It cannot know that this particular egg is slipping now, because the deployed policy never sees that signal.

That leaves two deployment choices: keep the sensor on the collection rig, or put it on every deployed robot.

Training-time touch can supply a class-level prior. It cannot catch a failure in the moment. For me, deciding between the two belongs in commissioning: does the cell need an alarm when something goes wrong, or only a better setpoint?

If the sensor stays on the deployed robot, response time helps decide which layer should read it. These are engineering bands to measure on the actual cell, not universal requirements.

I would treat these as starting estimates and measure the complete loop on the cell. Sampling, transport, inference, controller cycle and actuator response all consume the budget. A 60 Hz sensor begins with 16.7 milliseconds between frames before any of the other work happens.

If what you need is slip or over-force caught fast enough to prevent damage, put the trip beside the torque and workspace limits unless the complete learned loop has been shown to meet the requirement. A slower contact state may fit inside an action-policy cycle, but reported inference frequency alone does not commission that path.

A cell can need force sensing badly without feeding that signal into the foundation model.

What I would ask when specifying a cell

The first question I would ask is whether the robot can see the variable that decides success at the moment it matters.

If it can, do not add tactile input to the learned policy by default. Benchmark it against vision-only and require a gain on the task and failure metrics that matter. You may still need a force-torque sensor, motor-current limit or contact switch in the controller for quality or protection; that is a separate decision.

This is where my own test catches me: connector pins may be hidden because the gripper I chose is in front of them, which makes the occlusion a design choice rather than an intrinsic property of connector mating. A see-through finger, a camera in the palm, a different approach angle: any of those can move a task from the right column to the left. Before I conclude that a task needs another modality, I would ask whether it only needs a better view of the one it already has. I would try those changes before adding another sensing modality.

If it cannot, first ask whether a camera, fixture or approach change can reveal it. If not, choose the minimum measurement that exposes the state: wrist force-torque, motor current, a contact switch or fingertip tactile. Then decide whether the response belongs in a deterministic controller or the learned policy, how fast it must return, and how you will test sensor drift. The major general-purpose open checkpoints do not accept tactile input today, so a policy-level choice still means integration work and sensor-specific validation.

The tasks in that second group are not a research wish list. They are disproportionately the work the industry has struggled to automate for decades: mating connectors, routing harnesses, seating threads, judging by feel whether something has gone home. The connector that clicks is the tell. Somebody designed that click for a sense that is not sight. It does not prove touch belongs in a VLA, but it does identify the state the cell has to observe somehow. That is the same question I asked about every instrument in a plant: does the thing being measured determine the thing we care about? It is why I keep coming back to what a capture has to preserve before any of this gets trained.

One result would change my deployment default. If a policy can transfer force-sensitive control cleanly between a gel-shading sensor and a pin-displacement sensor on real tasks, without per-sensor calibration or fine-tuning, then run-time touch becomes much less bespoke. Shared tactile representations are already real; clean cross-sensor control is the threshold I have not yet seen crossed. If it has been, I would like to know.

Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA.