Hand-State-Conditioned VLA

Problem

In collaborative assembly, a robot has to decide both what to do and when to act. Pairing a VLA with action chunking learns collaborative manipulation end to end from demonstrations — but the difficulty comes from the chunking itself.

The policy predicts one future action chunk at a time and only replans after executing the first $T_a$ actions of it. If a chunk straddles the boundary between waiting and assisting in the demonstration, the robot reaches out before the human is ready. Prior work calls this demonstration action leakage and finds that it worsens as the execution horizon grows.

That makes $T_a$ a dial that directly governs collaboration quality:

• small $T_a$ → responsive, but jerky motion and higher inference cost
• large $T_a$ → smooth and efficient, but prone to acting too early

This thesis does not ask which point on that trade-off curve to pick. It asks whether the curve itself can be moved.

Three existing lines, each one step short

Approach

The main route is a low-dimensional state channel. A small projection network maps the hand-state vector into the model’s embedding space as one additional token, entering the sequence alongside the vision and language tokens. This follows how OpenVLA-OFT injects robot proprioceptive state, so the backbone is untouched.

Four arms under one protocol

This is the core of the evaluation: the three injection routes each correspond to the stance taken by a different published system, and they have never been measured side by side.

Task and data

The core task is a timing-sensitive object handover: the robot holds an object and waits while the collaborator reaches according to a randomised timing script covering early, on-time, late, feint, and retract-mid-reach.

These perturbations do two things. They create controlled ambiguity between observations that should mean wait and observations that should mean act. And they are the only way to separate a learned policy from a fixed distance threshold — a threshold cannot handle a retraction. A command-triggered grasping task serves as a control task.

Data is recorded at 10 Hz: multi-view RGB, robot proprioceptive state, actions, time-stamped hand state, and phase labels. Prior work reports that policies trained on a single demonstrator fall back on fixed routines when the collaborator changes, so training covers several collaborators and evaluation uses people not seen during training.

Perception pipeline

Hand keypoints are lifted to 3D using aligned depth and transformed into the robot base frame; hand velocity comes from position differences over a short buffer.

Model

The backbone is $\pi_{0.5}$ fine-tuned with LoRA. OpenVLA-OFT+ is used only for a small number of offline control points, to check whether the conclusions depend on a single architecture.

Core hypothesis: observation aliasing

Premature action may stem partly from observation aliasing near phase transitions: across the interval where waiting turns into assisting, visually similar RGB observations correspond to mutually incompatible target actions. A regression-based policy then compromises between those action labels, which shows up as reaching too early.

Hand state is a low-dimensional but temporally discriminative signal that may pull those two cases apart.

Three research questions

The three attribution controls

This is the part of the design I care about most, because it is an active attempt to disprove my own conclusion:

1. Waiting-data augmentation — add an equal amount of waiting footage to the baseline, to test whether the gain is only a shift in training distribution
2. Time-shuffled hand state — architecture and parameter count held exactly constant, hand state shuffled in time only, to test whether the gain depends on temporal structure or merely on extra parameters and static position
3. Scripted gating — a learning-free rule over hand distance, radial velocity, dwell time, and hysteresis, to test whether a rule is already enough. The timing-perturbation condition is what makes this control decisive, because a rule cannot handle a retraction

Criteria, fixed in advance

Support: at long horizons, fewer false commitments than the baseline, with a paired-difference confidence interval excluding zero; no increase in waiting drift or reaction latency; no drop in task success. The advantage should be larger under timing perturbation than under normal timing.

Falsification: the two paired curves are statistically indistinguishable; or waiting-data augmentation removes the gain; or representation analysis shows no reduction in observation aliasing.

Robustness requirement

When no reliable hand information is available, the added path must degrade to a no-op: hand state is randomly dropped during training alongside a dedicated missing-state embedding, and a detector-failure scenario verifies that the policy recovers baseline behaviour.

Metric definitions

Schedule

Experimental and writing phases overlap by design, and the decisive Go/No-Go test sits early.

Data protection and informed-consent procedures are confirmed with the supervisor before any human data is recorded.