SoftVTBench shows robot policies sometimes exceed deformation limits in successful runs

Most benchmarks for deformable-object manipulation check only whether a task got done, the authors argue, missing a separate problem: a policy can complete a task while still letting the object slip or squeezing it with excessive compression. They trace this blind spot to a specific gap in the field: there was no visuo-tactile dataset that pairs what a policy can see and feel during contact with an independent, ground-truth measurement of the object's physical state across a complete task.

To close that gap, the authors built SoftVTBench, a visuo-tactile dataset for physical-interaction-aware deformable-object manipulation. It contains 4,000 expert demonstrations across more than 50 assets, including volumetric deformable objects and rigid counterparts visually matched to look the same. Each episode is recorded at 20 Hz, synchronizing multi-view RGB video, dual-finger tactile RGB images and marker-motion signals, proprioception, language instructions, and both binary and continuous gripper actions. Alongside these policy-visible streams, every episode also carries evaluator-only finite-element (FEM) simulation states, an independent ground-truth reconstruction of the object's physical deformation that the policy itself never sees during the task.

Building on the dataset, the authors set up a closed-loop benchmark around a new metric, the Deformation-aware Success Rate (DSR). Using fixed, object-specific calibration, DSR counts a rollout as successful only if it both completes the task and keeps the object's peak normalized deformation within tolerance, folding physical-interaction quality directly into the success measure rather than scoring it separately.

Testing three policies, Diffusion Policy, π0.5, and FastWAM, the authors found that all 12 in-distribution configurations contained rollouts the plain task-success metric would call successful but that actually violated the deformation tolerance, with the share ranging from 0.7% to 24% of each configuration's successes. Under distribution shift, meaning the policies were tested on conditions that differ from training, the visuo-tactile variants achieved higher task success in all six policy-suite comparisons and a higher DSR in five of the six; tested in-distribution instead, the benefit of adding touch was mixed rather than consistent.

The authors conclude that giving a policy access to touch data does not by itself guarantee effective fusion of vision and touch: the benefit is conditional, showing up more reliably once conditions shift away from training rather than staying constant. They present SoftVTBench as a shared resource for studying not just whether a manipulation policy succeeds, but how it physically interacts with deformable objects, and when touch actually helps that interaction.

Key facts

SoftVTBench pairs 4,000 expert demonstrations across more than 50 assets, deformable objects plus visually matched rigid counterparts, with independent finite-element ground truth, recorded at 20 Hz alongside multi-view RGB, tactile RGB, proprioception, language, and gripper actions.
Its Deformation-aware Success Rate (DSR) metric, built on fixed, object-specific calibration, counts a rollout as successful only if it completes the task and keeps peak normalized deformation within tolerance.
Across Diffusion Policy, π0.5, and FastWAM, all 12 in-distribution configurations contained 'successful' rollouts that actually violated the deformation tolerance, ranging from 0.7% to 24% of each configuration's successes.
Under distribution shift, visuo-tactile policy variants beat their non-tactile counterparts in task success across all six policy-suite comparisons and in DSR in five of six; the in-distribution advantage was mixed.
The authors conclude that adding touch sensing does not by itself guarantee effective fusion between vision and touch.

Why it matters

Most benchmarks for deformable-object manipulation only check whether a task got done, so a policy that finishes by letting an object slip or squeezing it too hard looks identical to one that handles the object well. SoftVTBench and its Deformation-aware Success Rate close that gap: by pairing what a policy senses during contact with an independent, evaluator-only measurement of the object's physical state, they let researchers score interaction quality directly, and test when adding touch sensing actually helps rather than assuming it always does.

Who it affects

This is aimed at robotics researchers and engineers building or evaluating manipulation policies for deformable and soft objects, especially anyone working on visuo-tactile fusion for contact-rich tasks. It is most directly useful to teams already using or considering policies like Diffusion Policy or π0.5 for this kind of manipulation. The text names no individual authors, institutions, or affiliations behind the work.

How to use it

Researchers evaluate a manipulation policy on SoftVTBench by running it closed-loop in the environment and scoring rollouts with the Deformation-aware Success Rate instead of plain task success. DSR relies on fixed, object-specific calibration: a rollout only counts as a real success if the task is completed and the peak normalized deformation stays within that object's tolerance, with the finite-element simulation states acting as an independent, evaluator-only check the policy never has access to while it acts.

How solid is it

The result rests on a fairly broad empirical base: 4,000 demonstrations across more than 50 assets, three different policies (Diffusion Policy, π0.5, and FastWAM), 12 in-distribution configurations, and six policy-suite comparisons under distribution shift, with the pattern of hidden violations showing up in every configuration tested rather than in one isolated case. The source text gives no publication or submission date, and it does not describe the tactile-sensor hardware or robot platform used, or define exactly how 'peak normalized deformation' is computed beyond the term itself. On Hugging Face's papers board the submission has drawn light attention so far: 12 points and two comments, which reflects visibility rather than independent verification of the results.

Risks and caveats

The 0.7% to 24% range means the scale of the hidden-violation problem varies a lot by configuration, and the source gives no absolute DSR or task-success percentages for any policy, only these relative counts and the one range. The touch-sensing advantage is conditional, not universal: it shows up clearly once conditions shift away from training, but the authors call the in-distribution benefit 'mixed'. The source also does not say whether the SoftVTBench dataset, code, or benchmark suite is or will be released publicly, so it is not yet clear whether other teams can use it directly.

“These results show that making touch available does not by itself ensure effective multimodal fusion.”

— the paper's authors