Xilin Zhu
ContactAvailable for work
← All work

Master thesis · FZI Research Center for Information Technology

Hand-State-Conditioned VLA

Hand-State-Conditioned VLAs for Timing-Aware Human-Robot Collaboration

2026.09 — 2027.02In progress

This project is in progress and has no results yet. What follows is the research design.

Giving a collaborative robot a sense not only of what to do but of when to move — by injecting the collaborator's 3D hand position and velocity into a VLA policy as a low-dimensional state token, and testing whether that reduces premature action without shortening the execution horizon.

Thesis duration (weeks)
26
Go/No-Go decision (week)
10
Control arms under one protocol
4
Share of timing-perturbation trials
≥30%
Perception pipeline rate
≥10 Hz
Collaborators recorded
≥3

Problem

In collaborative assembly, a robot has to decide both what to do and when to act. Pairing a VLA with action chunking learns collaborative manipulation end to end from demonstrations — but the difficulty comes from the chunking itself.

The policy predicts one future action chunk at a time and only replans after executing the first TaT_a actions of it. If a chunk straddles the boundary between waiting and assisting in the demonstration, the robot reaches out before the human is ready. Prior work calls this demonstration action leakage and finds that it worsens as the execution horizon grows.

That makes TaT_a a dial that directly governs collaboration quality:

  • small TaT_a → responsive, but jerky motion and higher inference cost
  • large TaT_a → smooth and efficient, but prone to acting too early

This thesis does not ask which point on that trade-off curve to pick. It asks whether the curve itself can be moved.

Three existing lines, each one step short

Line of workWhat it achievesWhat is missing
Timing interventionInference-time guidance from end-effector position; adaptive horizons from internal attention weights; a reflective module that asks for helpNone conditions execution on the collaborator’s continuous state
Human cuesGaze and gestures determine which object or part to manipulate; hand pose used as an auxiliary prediction targetAddresses what to manipulate, not when to act
Human motion forecastingForecasts fed into a closed-loop MPC planner; hand motion used as a standalone handover-intent detectorPlanning already uses it, but when a VLA decides how long a chunk runs, the trigger still comes from inside the robot

Approach

The main route is a low-dimensional state channel. A small projection network maps the hand-state vector into the model’s embedding space as one additional token, entering the sequence alongside the vision and language tokens. This follows how OpenVLA-OFT injects robot proprioceptive state, so the backbone is untouched.

Four arms under one protocol

This is the core of the evaluation: the three injection routes each correspond to the stance taken by a different published system, and they have never been measured side by side.

ArmHow hand information is usedStance
State tokenLow-dimensional vector projected into one input tokenThis thesis
Pixel overlayHand skeleton drawn into the observation imageShows where the hand is, but cannot encode 3D velocity — and velocity is exactly what separates reaching from retracting
Auxiliary supervisionHand pose as an auxiliary prediction target, disabled at inferenceA training signal, not a policy input
No hand informationNot used at allBaseline

Task and data

The core task is a timing-sensitive object handover: the robot holds an object and waits while the collaborator reaches according to a randomised timing script covering early, on-time, late, feint, and retract-mid-reach.

These perturbations do two things. They create controlled ambiguity between observations that should mean wait and observations that should mean act. And they are the only way to separate a learned policy from a fixed distance threshold — a threshold cannot handle a retraction. A command-triggered grasping task serves as a control task.

Data is recorded at 10 Hz: multi-view RGB, robot proprioceptive state, actions, time-stamped hand state, and phase labels. Prior work reports that policies trained on a single demonstrator fall back on fixed routines when the collaborator changes, so training covers several collaborators and evaluation uses people not seen during training.

Perception pipeline

Hand keypoints are lifted to 3D using aligned depth and transformed into the robot base frame; hand velocity comes from position differences over a short buffer.

Model

The backbone is π0.5\pi_{0.5} fine-tuned with LoRA. OpenVLA-OFT+ is used only for a small number of offline control points, to check whether the conclusions depend on a single architecture.

Core hypothesis: observation aliasing

Premature action may stem partly from observation aliasing near phase transitions: across the interval where waiting turns into assisting, visually similar RGB observations correspond to mutually incompatible target actions. A regression-based policy then compromises between those action labels, which shows up as reaching too early.

Hand state is a low-dimensional but temporally discriminative signal that may pull those two cases apart.

Three research questions

RQQuestionExperiment
RQ1Does conditioning on hand state reduce premature action, and how does the effect vary with TaT_a?Paired execution-horizon sweep. TaT_a is an inference-time parameter, so sweeping it needs no retraining — two checkpoints cover the whole plot
RQ2How much does the way hand information is used matter?Four systems compared on the same data, backbone, and TaT_a
RQ3If there is an improvement, does it come from the information, the added parameters, or a rule that needs no learning?Three attribution controls

The three attribution controls

This is the part of the design I care about most, because it is an active attempt to disprove my own conclusion:

  1. Waiting-data augmentation — add an equal amount of waiting footage to the baseline, to test whether the gain is only a shift in training distribution
  2. Time-shuffled hand state — architecture and parameter count held exactly constant, hand state shuffled in time only, to test whether the gain depends on temporal structure or merely on extra parameters and static position
  3. Scripted gating — a learning-free rule over hand distance, radial velocity, dwell time, and hysteresis, to test whether a rule is already enough. The timing-perturbation condition is what makes this control decisive, because a rule cannot handle a retraction

Criteria, fixed in advance

Support: at long horizons, fewer false commitments than the baseline, with a paired-difference confidence interval excluding zero; no increase in waiting drift or reaction latency; no drop in task success. The advantage should be larger under timing perturbation than under normal timing.

Falsification: the two paired curves are statistically indistinguishable; or waiting-data augmentation removes the gain; or representation analysis shows no reduction in observation aliasing.

Robustness requirement

When no reliable hand information is available, the added path must degrade to a no-op: hand state is randomly dropped during training alongside a dedicated missing-state embedding, and a detector-failure scenario verifies that the policy recovers baseline behaviour.

Metric definitions

MetricDefinition
False commitmentA complete but premature object handover
Waiting driftPath length accumulated by the end effector during the waiting phase
Reaction latencyTime from the human’s readiness cue to the robot starting to move
Target selection accuracyCorrectness on the which object dimension

Schedule

Experimental and writing phases overlap by design, and the decisive Go/No-Go test sits early.

WorkWeeks
Hand-state pipeline and baseline1–5
Data collection3–7
Decisive execution-horizon sweep6–10
Offline observation-aliasing analysis8–12
Attribution controls and injection ablation9–14
Informal pilot runs and detector-failure check13–20
Thesis writing1–23
Final revision and supervisor feedback21–26

Data protection and informed-consent procedures are confirmed with the supervisor before any human data is recorded.