A Unified Latent Action Policy for Cross‑Embodiment Manipulation

A continuous latent, distilled then decoded

UniLAP learns a shared latent policy from action‑free video across embodiments. Three components train in two stages: a continuous flow‑matching latent policy, a label‑free motion‑supervision target, and a per‑embodiment action decoder.

Prior pipelines quantize the latent into a discrete codebook; UniLAP models it as a continuous flow, preserving the multimodal distribution manipulation requires, and acting without future frames at inference.

A real xArm7, three tabletop tasks

We deploy on a 7‑DoF UFACTORY xArm7 with a single fixed third‑person ZED 2i view, comparing against π_0.5 on three tabletop tasks. Following our recipe, we collect 50 teleoperated demonstrations per task, fit an 8‑DoF joint‑space decoder from scratch, and finetune the latent policy. The policy runs chunked control through a two‑process ZMQ client–server loop (observation in, a 16‑step joint chunk out), with no inverse kinematics or camera calibration.

Successful rollouts · buttons switch the object’s initial position

Deploy: NVIDIA RTX PRO 5000 Laptop GPU · 4 flow‑matching steps · 16‑step open‑loop horizon. Latency is task‑independent across all rollouts.

Cross‑embodiment adaptation from a shared latent

The same action‑free pretraining is shared across embodiments; each one adds only a fresh action decoder and a light policy finetune, no per‑robot retraining. At 1.1B parameters, UniLAP compares favorably with latent‑action and VLA baselines up to 7B.

What the latent captures

Two probes indicate the latent encodes action, not appearance: it reconstructs the future from the current frame alone, and it tracks what an instruction asks and where, not its surface wording.

BibTeX

@article{unilap2026,
  title   = {UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation},
  author  = {Lei Yao and Qiao Feng and Mutian Tong and Qihao Yun and Lap-Pui Chau and Lingjie Liu},
  journal = {Under Review},
  year    = {2026},
}

Citation venue and key are placeholders. Replace with the final venue and key on publication.

UniLAP builds on open datasets, simulators, and pretrained models. The dataset frames and simulator renders shown on this page come from the sources below and remain under their original licenses and terms; the real-robot xArm7 footage and the figures are our own.

• NVIDIA DreamDojo provides the frozen continuous LAM teacher that Stage I distills.
• NVIDIA Isaac GR00T provides the LeRobot data loader and the GR1 humanoid corpora used in Stage-I pretraining.
• Apple EgoDex provides the egocentric human video used in Stage-I pretraining and for the hand-pose forecasting benchmark; the egocentric frames shown here are from that dataset.
• LeRobot provides the v2.0 dataset format used throughout, and DINOv2, SigLIP 2, and Flan-T5 provide the frozen backbones.
• SimplerEnv (with ManiSkill2-real2sim) provides the simulated evaluation benchmark for the WidowX and Google Robot setups; the single-arm simulation clips here are our own policy rollouts rendered in it. The matching training data are BridgeData V2 and the RT-1 corpus, used via the LeRobot v2.0 conversions from IPEC-COMMUNITY.
• RoboCasa and robosuite provide the closed-loop evaluation environments for the GR1 humanoid tasks (via the robocasa-gr1-tabletop-tasks suite), and the RoboCasa365 pretrain split is part of Stage-I co-training; the humanoid clips here are our own policy rollouts rendered in those environments.