VLA-Tug: Anticipating When and How Vision-Language-Action Models Fail via Calibrated Latent Probing

Anonymous authors

Paper under review

Paper arXiv soon Demo Code Data soon BibTeX

Anticipate the failure. Name its mode. Bound the errors.

A small probe reads the latent layers of a Vision-Language-Action model while it acts. It raises an alarm before the failure, predicts how the policy fails, and its alarm threshold is calibrated to hold false-alarm and missed-detection targets.

Overview of VLA-Tug: the probe on the VLA's latent layers, its three heads, and the calibrated alarm. — **Overview.** We introduce VLA-Tug, a probe on the VLA backbone embeddings that predicts failures and their mode. (a) A recurrent encoder maps the embedding into a latent failure space; a score head and a calibrated trigger raise the alarm, and a mode head then predicts the failure mode. (b) VLA-Tug alarms before the failure, allowing early stopping or intervention by an expert model.

Interactive demo

Choose a rollout of an unseen task and play it. The probe's state, the scores of three methods and the predicted failure mode follow the video.

The demo could not load its data. Serve this folder over http (for example python -m http.server in website/) instead of opening the file directly.

(a) VLA-Tug: from the backbone embedding to the alarm and the predicted failure mode

Agent view

Wrist view

Instruction

VLA

Backbone

Backbone embedding

Action expert

Action chunk

VLA-Tug

Latent state encoderLSTM

Latent failure space

Failure score headMLP

Calibrated trigger

No alarm

Failure mode headMLP

Predicted failure mode top 3

VLA-Tug probes the VLA's backbone embedding and maps it into a latent failure space.

Expert modelearly stop or intervention

current step VLA-Tug alarm × annotated failure time

(b) Runtime failure detection

Agent view

Wrist view

alarm dashed line: calibrated threshold annotated failure time and the on-time window before it

Abstract

Vision-Language-Action models (VLAs) enable general-purpose embodied intelligence, but their performance is brittle under distribution shift, where unanticipated failures can damage equipment or endanger people. Deploying VLAs in real-world systems therefore requires a runtime monitor that is timely, calibrated, and informative while being efficient enough to run on the edge. Existing methods fail to fulfill these requirements. We present VLA-Tug, a lightweight probe over the latent layers of a VLA backbone that improves early failure detection and bounds the missed-detection and false-alarm rates, with minimal inference overhead. We show that supervising on the pre-failure segment of a trajectory increases early failure detection at matched false-positive rates by up to 30% over the strongest learned baseline. Under shift to unseen tasks, the bounds on missed-detection and false-alarm rates remain valid in up to 86% of runs, versus at most 21% for baselines. Unlike existing methods, VLA-Tug predicts the failure mode to enable corrective actions, raising top-3 accuracy on unseen tasks by up to 44% over a majority baseline. We release a dataset of 3,180 annotated manipulation episodes to support future work on latent failure detection in VLAs.

Probe the latent layers. Calibrate the alarm.

Pool

Average the backbone embedding of every policy step over the agent-image, wrist-image and language tokens: a 6144-d vector per step.

Train

Train a probe on the steps that precede a failure: an LSTM over the embeddings with a failure-score head, a failure-mode head and a semantic (CLIP) head.

Calibrate

Set the alarm threshold on held-out rollouts with max-conformal prediction or Learn-then-Test, which bounds the false-positive rate and the early true-positive rate together.

Evaluate

Score against annotated failure times: the area under the early true-positive rate versus false-positive rate curve up to 40%.

Results

TODO: one sentence on the headline result. Numbers must match the paper's tables.

Early failure detection on seen and unseen tasks: the area under the early-TPR vs. FPR curve up to an FPR of 40% (in %, higher is better), the mean over the runs of a setting, with the standard deviation over them after the ±; the number of runs and a qualitative difficulty (more stars, harder) are under the name of the setting (seeds; LIBERO-X skill: one fold per held-out skill; real robot: 5 seeds, each with one fold per held-out task, 8 folds). Seen: held-out episodes of training tasks; unseen: held-out tasks. Cal.: calibrator (MC max-conformal, FC functional conformal). Cost on LIBERO-X (lower is better): fit time, memory to score a new episode, and per-step latency on a GPU and a CPU. Dashes: not available. Best in bold, second best in italics.
Method	Cal.	LIBERO-10 π0.5		LIBERO-10 GR00T		LIBERO-X task		LIBERO-X signature		LIBERO-X skill		Real robot		Cost (LIBERO-X)
		3 seeds		3 seeds		3 seeds		3 seeds		8 folds		5 seeds × 8 folds
		difficulty *		difficulty *		difficulty *		difficulty **		difficulty ***
		Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Fit (s)	State (MB)	GPU (ms)	CPU (ms)
SAFE-LSTM	FC	75.5±13.4	34.9±3.5	74.1±13.0	51.2±8.4	64.6±3.5	40.4±2.3	64.7±7.4	41.2±7.1	62.5±2.4	30.1±8.5	44.5±13.5	25.1±18.8	43.5	26.2	0.066	0.86
SAFE-MLP	FC	74.1±2.0	39.9±15.1	62.9±18.2	39.2±9.4	59.3±2.0	42.3±2.9	53.7±2.4	46.2±2.3	56.8±6.0	23.5±12.4	40.1±15.9	20.2±16.6	114.5	6.3	0.034	0.032
STAC	MC	33.8±5.1	36.4±5.3	24.2±7.5	17.5±9.8	21.0±2.3	28.0±0.1	25.6±3.7	28.7±0.3	24.6±5.0	21.5±8.0	–	–	0.0	0.0	21.30	–
ActProbe	MC	39.3±11.3	26.7±2.6	32.4±16.8	30.1±9.3	12.6±4.2	22.7±4.6	11.9±3.1	13.9±1.5	20.3±9.4	13.6±7.1	–	–	7.1	0.09	0.15	0.11
FailureSpot	FC	61.9±3.0	36.2±21.0	36.9±19.8	44.7±18.6	26.3±2.1	20.4±1.2	28.0±2.5	26.2±5.6	30.4±3.4	19.6±7.9	–	–	9.2	6.4	0.033	0.029
Mahalanobis	FC	85.7±4.1	40.2±15.5	70.7±6.4	29.2±8.7	69.6±6.2	30.6±2.2	64.4±3.0	22.3±4.0	71.2±3.1	15.9±9.6	43.0±16.5	19.6±2.5	15.2	302.0	0.41	5.37
Euclidean	FC	73.1±4.4	24.2±10.6	56.0±6.8	23.1±11.4	70.5±4.2	28.5±3.0	65.4±2.1	19.7±0.8	68.5±4.8	19.0±6.5	45.1±14.2	8.3±12.8	0.4	3056	13.19	368.83
PCA-KMeans	FC	35.7±12.2	12.9±5.1	44.5±8.7	20.0±0.0	31.7±3.4	25.0±1.2	35.3±1.8	32.5±1.7	37.8±7.7	20.7±7.1	24.0±11.4	19.4±7.7	39.9	1.6	0.12	0.055
VLA-Tug (ours)	MC	75.1±12.5	47.9±20.9	70.5±10.4	47.4±13.5	67.9±5.0	40.2±1.2	67.0±6.6	47.0±0.6	67.8±4.8	39.0±7.0	47.6±10.9	25.8±11.6	6.8	26.5	0.12	0.94

Dataset

Pooled embeddings and annotations of simulated rollouts of two VLAs (GR00T-N1.7 and π0.5) on LIBERO-10 with occlusions and on LIBERO-X, with failure times and failure modes. Dataset: link soon.

Code and reproduction

The code covers the probe, the calibration (SAFE band, max-conformal, Learn-then-Test), the baselines, the metrics and the scripts that produce the paper's tables and figures.

pip install -e ".[dev]"
python -m cli.download liberox_groot       # pooled features, targets, manifest, splits
python -m cli.train --datasets liberox_groot
python -m cli.evaluate --datasets liberox_groot
python -m cli.figures                      # main table, ablations, curves

These commands are taken from the repository README. During review the code is at the anonymous mirror linked above.

BibTeX

@misc{vla-tug,
  title  = {VLA-Tug: Anticipating When and How Vision-Language-Action Models Fail via Calibrated Latent Probing},
  author = {Anonymous},
  year   = {2026},
  note   = {Under review}
}

TODO: replace with the final entry (authors, arXiv id) once the paper is public.

VLA-Tug: VLATug Anticipating When and How Vision-Language-Action Models Fail via Calibrated Latent Probing