VLA Vision-Language-Action model: a robot policy that starts from a pretrained VLM backbone and adapts it to generate actions from visual observations and language instructions. Large-scale VLM pretraining is a core part of the recipe. See Pi-0 and GR00T N1.
WAM World-Action Model: a policy that starts from a pretrained world-model or video backbone and adapts it to represent or predict how the scene changes over time and emit corresponding actions. We use WAM as the term throughout this post.
VLM Vision-Language Model: a model pretrained on image-text or video-text data to produce language outputs grounded in visual inputs, usually before being adapted for robot control.
Video backbone A pretrained video model reused as the central representation or generator inside a robot policy.
World model A model that predicts a future world state, conditioned on some action abstraction such as language, robot actions, or latent actions. The predicted state may be represented as images, video, point tracks, object states, or latent features. See the classic World Models paper and NVIDIA’s Cosmos world foundation model paper.
Grounding Connecting symbols (e.g. words in a language instruction) to the perceptual and motor referents that satisfy them. Language-to-action grounding in particular means turning an instruction like “pick up the red mug” into the visual percepts and motor commands that actually accomplish it. The grounding gap is the persistent shortfall between what a model knows about language and what it can reliably cause to happen in the physical world.
Inverse dynamics Given a current observation ot and a future observation ot+k, infer the most plausible action or action sequence that would produce the transition.
Joint prediction Given ot and language lt, train one policy π(ot, lt) to predict both future observations ot+1:t+k and actions at:t+k.
Action chunk A short horizon action sequence at:t+k — i.e. the k actions at, at+1, …, at+k−1 — such as joint commands, end-effector deltas, and gripper states, predicted in one policy call. See ACT and Diffusion Policy.
Mixture-of-Transformers (MoT) Several modality-specific transformers or experts, such as a video transformer and an action transformer, connected through shared attention while keeping separate weights. See the related Transfusion paper.
Diffusion Transformer (DiT) A transformer backbone used inside diffusion or flow-matching models to denoise image, video, or action tokens over multiple steps. DiT commonly uses adaptive layer normalization (adaLN) to inject timestep conditioning into transformer blocks. See the Peebles and Xie DiT paper.
VAE Variational Autoencoder: in this post, mainly image and video VAEs that compress high-resolution images or videos into latent representations before generation or policy learning. This reduces token count substantially; for example, Wan 2.1’s VAE uses 4× temporal and 8×8 spatial compression, while Wan 2.2-5B uses a higher-compression 4× temporal and 16×16 spatial interface. See the original VAE paper, Rombach et al.’s latent diffusion paper, the Wan paper, and the Wan 2.2 release.
Wan A family of large pretrained video-generation models often used as the video backbone in recent WAMs. See the Wan paper.
Cosmos NVIDIA’s world foundation model family for physical AI, including video prediction models that can be adapted for robotics and policy learning. See the Cosmos paper.
DROID Distributed Robot Interaction Dataset: a large real-world manipulation dataset with more than 50k demonstrations across varied tasks, collected using Franka Panda robot arms. See the DROID paper.
RoboArena A distributed real-world benchmark for evaluating generalist robot policies on open-ended language-conditioned tasks. See the RoboArena paper.
RoboLab A high-fidelity simulation benchmark for analyzing task-generalist robot policies across visual, relational, and procedural competencies. See the RoboLab paper.
CALVIN A language-conditioned manipulation benchmark focused on long-horizon task sequences in simulation. See the CALVIN paper.
LIBERO A robot-learning benchmark for studying knowledge transfer, lifelong learning, and generalization in manipulation. See the LIBERO paper.
RoboTwin A simulation data generator and benchmark for robust bimanual robotic manipulation under domain randomization. See the RoboTwin 2.0 paper.
FAST / BEAST Discrete action-tokenization methods that turn continuous robot actions into token sequences, making action learning more compatible with VLM-style training. See the FAST paper and BEAST paper.
VPP Video Prediction Policy: a WAM-style method that uses predictive visual representations from a video model to condition robot actions. See the VPP paper.
LAPA Latent Action Pretraining from Videos: a method for learning action-like latent variables from videos without ground-truth robot action labels. See the LAPA paper.
OOD Out-of-distribution: a task, object, environment, or instruction outside the examples used during training or demonstration.
FLOP / ZFLOP Floating-point operations measure training compute. 1 ZFLOP equals 1021 FLOPs.
H100 / GPU-hour H100 is a high-end NVIDIA training GPU. A GPU-hour means one GPU running for one hour, a rough unit for comparing training cost.
BF16 Brain floating point 16-bit: a lower-precision number format commonly used to train large neural networks efficiently.
I2V Image-to-video: a video-generation setup conditioned on an initial image or frame.
Background: two building blocks. A visuomotor policy maps current observations plus a goal or instruction to robot actions. A world model predicts future visual or latent states from the current state plus an action or goal abstraction. A WAM sits at the overlap: it leverages a pretrained video/world-model backbone as a prior and predicts both future states and robot actions.
Visuomotor policy: language instruction and current observation in, action sequence out.
World model: current world state plus an action abstraction in, future image or latent out.
Introduction
Last year, my Scholar Inbox digest was dominated almost every day by new VLA papers. This changed in the last months, and a different keyword is coming up almost daily now too: WAM, short for World-Action Model. In October 2025, I wrote in my State of VLA post that WAMs were a small subfield within VLA research and far less popular than VLAs initialized from VLMs [60]. That has changed fast, and my wish to see more work in this direction has already become reality.
So what changed, and why now? Maybe it is because WAMs are the shiny new thing everyone wants to work on, or VLA authors ran out of new names for their own VLAs, since basically all “-VLA” names like “X-VLA” and “Ego-VLA” are already used. So now we can recycle them for the WAM area. But more likely it has something to do with VLM-based VLAs getting stuck. Modern VLAs benefited from massive vision-language pretraining, but they still hit a language-to-action grounding wall. The problem of mapping language and pixels into behavior still has to be learned from robot data. WAMs offer a different starting point. They use pretrained video or world-model backbones that already model how scene dynamics change under language conditioning. If that prior transfers to behavior generation, the remaining video-to-action gap may be smaller than learning language-to-action grounding directly.
But the ideas behind WAMs are not new. Early WAMs like UniPi [10] proposed essentially this approach back in 2023. So why did it take several years for the paradigm to enter the robot foundation model mainstream, and where does it actually stand today? This post takes a closer look at the modern WAM landscape to answer the central question:
Central question: Is this a real paradigm shift in research and industry, or just a short hype cycle? And if the recipe works so well, why did it take several years after early papers like UniPi for WAMs to become so popular?
My take: WAMs will become the second major recipe for robot foundation models, alongside VLM-based VLAs. The open questions are which formulation of them wins, and which parts of the model architecture and pipeline actually matter. It is likely that the winner is neither pure VLA nor pure WAM, but a hybrid of both.
This is my map of the modern WAM space: how to categorize and understand WAMs, what changed since the early models, and how current results compare to VLAs. For a broader survey, see the recent NTU survey “World Model for Robot Learning: A Comprehensive Survey” [57], which maps world models for robot learning across simulation, evaluation, navigation, and autonomous driving.
Table of contents
- The Two Representation Bets for Generalist Policies
- Why World-Action Models? Our Hypotheses
- Understanding Modern WAMs: Core Formulations
- Practical Considerations
- Why the Modern VLA Baseline Still Matters
- Are the Two Representation Roads Actually One?
- Closing Thoughts
- Sources
The two representation bets for generalist policies
The field currently has two major representation bets for robot foundation models in both research and industry. Many teams are building on the traditional VLA recipe established by Pi-0 [2] and later refined by Pi-0.5 [4], using VLM backbones as the starting point for policy learning. This VLM-backbone recipe appears in public work from teams including NVIDIA GR00T [5], Xiaomi Robotics [27], Being-H0.5 [28], and others.
More recently, a different paradigm has emerged: using pretrained video backbones as an alternative path toward generalist manipulation. Public examples now span NVIDIA’s DreamZero [8] and Cosmos Policy [13], Ant Group’s LingBot-VA [9], Rhoda AI’s DVA [40], Sereact’s Cortex 2.0 [45], and Mimic Robotics with mimic-video [14]. At the same time, many university labs and open research groups are also pushing the frontier with new ideas, including Video Prediction Policy [24], Unified Video Action Model [39], and Fast-WAM [23]. We discuss these in more detail below.
The choice of backbone impacts the full training and evaluation pipeline, from training recipe and data mixture to inference optimizations. Given the cost of running these models at scale, most teams will likely have to prioritize one direction (VLA or WAM) first rather than fully pursuing both in parallel. Which path proves out, or whether the two converge, is still open. Which one would you bet on today? In the following sections, we dive deeper into both sides of this decision.
Why World-Action Models? Our hypotheses
Before we dive deeper into current models, let’s first review why WAMs are attractive as an alternative to VLM-based VLAs. It also helps to first place WAMs inside the broader landscape of world models in robotics.
The grounding gap
To understand why WAMs are attractive, it helps to understand the core challenge of “classical” VLAs built on VLM backbones. The motivation for the first VLAs was to leverage the internet-scale knowledge of VLMs for robotics. VLMs are trained on massive amounts of vision-text data and show notable zero-shot performance on many vision tasks. The VLA recipe then adapts these pretrained representations for action generation.
However, there is a major domain gap between VLM pretraining and embodied manipulation. Several VLA papers either observe degradation of pretrained VLM capabilities or design around it, particularly when the action-learning objective diverges sharply from the original VLM objective. VLM2VLA frames this directly as catastrophic forgetting during the VLM-to-VLA transition [55]. Knowledge Insulation reports similar findings and makes the concern architectural: it isolates the gradients of the flow-matching action expert from the VLM backbone to preserve pretrained language/vision knowledge, improving training convergence, task performance, and language following [20]. Recent solutions like VLM co-training and discrete action tokenizers have helped, but the core challenge remains: grounding language into physical action from limited robot data. We cover these solutions in the modern VLA baseline section below.
This naturally raises the question: what if we started from a backbone that already represents how language maps to visual change in the world?
Core hypotheses for WAMs as policy representations
The core idea is simple: instead of using a VLM backbone to jump-start imitation learning, use a pretrained video backbone. Current video models are trained on large video corpora and learn spatiotemporal representations of how visual scenes evolve. Crucially, current video models are often text-conditioned: they are trained to generate videos from precise language descriptions, sometimes with a reference frame and sometimes from text alone. Many of these videos contain intentional behavior: hands reaching, tools moving, objects being manipulated, and scenes changing because someone or something acted. That makes video backbones attractive as a model prior for generalist manipulation. Before seeing any robot actions, the backbone already encodes useful links between language, visual change, and plausible object interactions. The Veo 3.1 demonstration below is a quick illustration.
I would treat the next three points as hypotheses, not conclusions. They are recurring claims across papers, discussions with peers, and my own read of the field, supported by qualitative intuition, simulation evidence, and a few early real-world signals, but not by clean matched comparisons yet:
- Predicting future world changes correlates with generating the necessary actions. Inverse dynamics prediction is often easier than pure action generation [26]. If the desired outcome is known, inferring the action that produced it is usually simpler than predicting the action directly from the instruction and current observation. Pi-0.7’s visual-subgoal results point in the same direction: when the policy is given a desired future image, action prediction becomes more direct and training converges faster [43].
- Video pretraining provides grounding between language and physical change. Video models learn to map text descriptions to visual outcomes. If this transfers to robotics, it could reduce the amount of grounding that has to be learned from robot demonstrations alone.
- Video data regularizes robot policies. Robot datasets are small relative to web-scale video. Either through pretraining on video first or through co-training on video alongside robot data, the broader visual prior can reduce overfitting; the benefit depends on the dataset, objective, and architecture. DreamZero [8] and Fast-WAM [23] both show that, during robot fine-tuning, WAMs perform best when action learning is co-trained with a video-prediction objective.
A quick experiment: how much does a frontier video model already “understand” about robot manipulation?
How much do modern video models already capture before any robotics-specific action head is added? We ran a simple experiment with Google’s Veo 3.1, a frontier video generation model. Given a single context frame from an original RoboArena rollout of a toaster task in the DROID setup, we prompted Veo to push the toaster lever (the reference task, matching the original DROID demonstration) and then pick up an orange sitting to the left (the composed extension, beyond the demonstration). This video is very unlikely to be part of Veo’s pretraining data, but we cannot verify the training set directly; treat this as a qualitative check of the prior, not a controlled probe of training-set membership. One-shot attempt, no prompt optimization.
The prompt used was:
“Given this initial frame, generate a video of the robot arm pushing the toaster lever. After finishing that task, the robot should pick up the orange on the left side of the toaster and stop after it has picked it up.”
Context frame and ground-truth rollout:
Veo 3.1 generated rollouts (zero-shot, no robotics fine-tuning):
The generated rollout is surprisingly good for a model that was not explicitly trained as a robot policy. The generated motions are smooth, the background remains stable and consistent, and the robot follows a plausible trajectory toward both target objects. Even the sequencing is respected: finish the lever, then move to the orange.
The limitations are equally visible: The model does not fully push the toaster lever down and at points appears to attempt the opposite motion (pulling it up). More visibly, the pinch gripper from the original DROID setup morphs into a four-fingered hand. The fixed-base robot arm is reimagined, almost instantly after the context frame, as a different robot with fewer degrees of freedom. These artifacts are consistent with the model using broad visual priors rather than faithfully modeling the specific hardware.
Still, the result illustrates why video backbones are attractive for robotics: the model has a useful prior for what robot-object interaction should look like, even though it is not yet reliable enough for control. WAM fine-tuning is the attempt to turn that zero-shot imagination into reliable control.
Understanding modern WAMs: Core formulations
After establishing the core motivation, we can now focus on the current WAM research. In contrast to VLM-based VLAs, where the training recipe has largely converged around VLM co-training with a flow transformer for action generation, WAMs are still splitting into several active formulations. This is exactly what makes the area interesting right now: the field does not yet know which combination of design choices will win, or whether the best systems will merge parts of several.
To make the design space readable, we organize WAMs along three axes (which are not fully independent):
- Paradigm: what does the model predict, and how is the predicted video used to generate actions? (inverse dynamics vs joint prediction vs representation-only)
- Action integration: how do actions actually enter the model? (default action tokens vs action-as-image vs latent actions/plans)
- Architecture: how are the components composed? (Mixture-of-Transformers vs monolithic vs hierarchical)
The axes are not fully independent, and some WAMs do not fit well into a single category. I would not treat this as a perfect taxonomy. It should be more a practical map for reading the current papers without getting lost in naming choices. For each axis, I present the idea with an older paper and then a modern scaled-up version of the same rough recipe.
Paradigm: What the model predicts
The first axis is the policy formulation: what the model predicts, and how the predicted video is used to generate actions. Across modern WAMs, we see three directions that differ at the inference boundary: inverse dynamics, joint prediction, and representation-only.
Inverse dynamics: Predict the future, then infer the action
The inverse-dynamics setup is the easiest WAM recipe to understand: first imagine the future, then predict the most likely action from the video. This shifts the hard language-grounding problem into the video stage: translate the command into a plausible visual change. The bet is that video pretraining has already learned a useful part of this language-to-visual-change mapping, so the action head does not have to learn everything from robot demos and can focus on the inverse-dynamics problem instead.
UniPi [10] is a pioneering paper for this direction. It was probably the first modern execution of this recipe that was clearly aware of the potential of video diffusion for robotics: use video as a high-level plan, then use inverse dynamics to recover the low-level controls. In hindsight. A lot of recent WAM work looks like an improved version of this.
UniPi also shows why it took several more years until WAMs hit the mainstream. it used a CNN-based video diffusion stack from the Imagen Video era [56], and the video generator had to be pretrained from scratch. Our rough estimate in Footnote 2 puts that pretraining around 167 ZFLOPs, well outside the budget of most robotics labs. While this recipe existed before, it was not really reproducible for the average lab Modern inverse-dynamics WAMs can now sidestep this by starting from open DiT-based video backbones and fine-tuning them2.
![Figure 11. LingBot-VA architecture: inverse dynamics action prediction conditioned on video rollouts from a fine-tuned Wan 2.2-5B backbone. Image from Li et al., 2026 [9].](https://developer-blogs.nvidia.com/wp-content/uploads/2026/06/LingBot-VA-architecture.webp)
A modern version of this direction is LingBot-VA [9]. It turns Wan 2.2-5B into a robot video-action model through 16k hours of cross-embodiment pretraining. The important difference from UniPi is not just scale. LingBot-VA is causal and trained on long visual histories for closed-loop rollouts rather than open-loop video generation. It also uses a Mixture-of-Transformers (MoT) architecture: separate experts for video and action, each with its own weights, coupled through shared self-attention in each layer.
| Design choice | UniPi [10] | LingBot-VA [9] |
|---|---|---|
| Main idea | Generate a future video plan, then recover actions with inverse dynamics. | Fine-tune a video backbone for closed-loop robot world-action rollouts. |
| Backbone | CNN-based video diffusion (cascaded U-Net), trained from scratch in the Imagen Video era. | Wan 2.2-5B latent DiT (open weights). |
| Latent video VAE | None; generates low-resolution RGB futures. | Wan 2.2-5B (16×16 spatial, 4× temporal) [56]. |
| Action expert | Separate CNN action head. | MoT action expert coupled via joint attention. |
| Action-video coupling | One-way: video first, then actions. | Two-way: video conditions actions; generated actions condition video. |
| Robot training scale | Small, demonstration-only. | 16k hours of robot world-action pretraining across embodiments. |
There are several variants around the same theme. Video Prediction Policy [24], DiT4DiT [37], and mimic-video [14] do not necessarily need the final RGB video; they use intermediate video-model features as the predictive plan for an action decoder. DVA [42] and LingBot-VA lean more directly on generated or predicted future rollouts. The difficult part is that most papers change the video backbone, use different amounts of large-scale pretraining, tune different hyperparameters, and evaluate on different setups.
Joint prediction: Learn video and actions together
The second formulation is joint prediction. Instead of generating a future video first and then decoding actions, the model predicts video and actions together. This is the more coupled version of the WAM idea: the model is forced to learn what should happen and how to make it happen in the same prediction step.
![Figure 13. GR-1 architecture. Stage 1 pretrains on video prediction; stage 2 fine-tunes on robot data with a joint objective over future frames and action chunks. Image from Wu et al., 2023 [11].](https://developer-blogs.nvidia.com/wp-content/uploads/2026/06/GR-1-architecture.webp)
GR-1 [11] is an early foundation paper for this direction. It pretrained on large-scale video and then fine-tuned on local robot datasets with both video and action supervision. It used a GPT-2-style transformer policy, pretrained it on internet video prediction with readout tokens, and then fine-tuned it on robotics data with a joint video-action objective. Earlier works like R3M [15] and Voltron [16] had already shown that video and language can help representation learning for robotics, but GR-1 made a simple and important shift: it used video to learn better policy representations, not only image-level visual representations.
At the time, the CALVIN results were useful simulation evidence. On the harder ABC→D split, prior methods in the GR-1 table stayed below an average sequence length of 1.0, while GR-1 reached 3.06/5. This result is useful here because it makes the generalization signal easiest to read. By 2026 this number is outdated, but I still think the result matters historically. It showed that predicting future visual states could shape a better policy representation, not just a better visual encoder.
![Figure 15. GR-1 CALVIN ABC→D result summarized as average completed subtasks out of five. Values are redrawn from Wu et al., 2023 [11].](https://developer-blogs.nvidia.com/wp-content/uploads/2026/06/gr1_calvin_abcd_average_sequence_length-1.webp)
Wu et al., 2023 [11] and Xiaomi Robotics, 2026 [27].
DreamZero [8] is the modern scaled version of this idea. Instead of training a smaller transformer-style policy around a video-prediction head, it starts from Wan 2.1-I2V-14B-480P and turns the video diffusion backbone into a joint world-action model. The model denoises video and action tokens together inside one monolithic DiT. There is no separate inverse-dynamics module: action is another generated modality inside the same denoising process.
![Figure 15. DreamZero architecture. A single monolithic transformer, initialized from a 14B Wan video diffusion backbone, denoises video tokens and action tokens jointly. Image from Ye et al., 2026 [8].](https://developer-blogs.nvidia.com/wp-content/uploads/2026/06/DreamZero-architecture.webp)
DreamZero’s reported RoboArena score is an important real-world signal for WAMs. While most papers still focus on popular benchmarks like LIBERO and other simulation benchmarks, RoboArena is one of the few public real-world, open-ended evaluations, which is what makes the snapshot below worth pausing on.
In the April 2026 snapshot above, DreamZero reaches 1750 compared to 1622 for Pi-0.5, which is a meaningful signal for the potential of WAMs. This is not proof that WAMs are the better default, but it is a positive signal for their potential. It is interesting because DreamZero was trained only on DROID without an additional large-scale cross-embodiment robot training stage.
| Design choice | GR-1 [11] | DreamZero [8] |
|---|---|---|
| Main idea | Use future-frame prediction as an auxiliary objective while learning actions. | Denoise future video and robot actions together in one video diffusion backbone. |
| Backbone | GPT-2-style transformer policy with video-prediction readout tokens. | Wan 2.1-I2V-14B-480P video diffusion model adapted for robot control. |
| Scale | ~21M policy parameters; pretrained visual and language encoders kept separate [55]. | 14B Wan backbone, action-tuned end-to-end. |
| Generative objective | L2 reconstruction for future video and actions. | Flow/denoising for joint future-video and action generation. |
| Latent video VAE | None; pretrained MAE/ViT visual features. | Inherited Wan latent video VAE. |
| Language conditioning | CLIP. | T5-family text encoder (inherited from Wan). |
GR-1 showed the policy-level version of joint video-action prediction, while DreamZero combines the idea with a modern video-foundation-model and flow-matching setup. The core joint-prediction idea is the same as GR-1, but DreamZero changes nearly everything else around it, so this is far from a clean comparison.
GR-2 [12], Seer [29], PAD [57], UWM [58], UVA [41], and DreamVLA [40] sit around this broader joint-prediction wave. PAD was another early attempt for joint future image prediction and robot action generation inside one joint denoising process. UWM uses independent noise for video and action to support more flexible inference modes inside a joint transformer.
Representation-only: Skip video generation at inference
The third option is to use the video backbone purely as a representation and skip video generation at inference entirely. Fast-WAM is a nice example of this idea.
Fast-WAM [23] uses a similar Wan/MoT-style setup to LingBot-VA and closely matches its performance on simulated benchmarks even without 16k hours of large-scale robot pretraining. In addition, skipping video generation at test time makes its inference several times faster. However, Fast-WAM is one of the few public pieces of evidence for the representation-only hypothesis, and current simulation evidence is not enough to really convince me of this idea. But I am happy to be convinced in future work.
Most WAMs today keep some form of video generation at inference and are very slow. Faster WAMs like Fast-WAM will become a much bigger research area in the future.
Action integration: How actions enter the model
After discussing how to combine video and action prediction, let’s focus on how actions are represented inside the model. The choice is important because the pretrained backbone knows how to denoise visual tokens, not continuous robot actions, so there is a real modality mismatch. I see three variants in the current papers.
Default action tokens
The simplest default is to add action tokens, continuous or discrete, and an action head, where actions are treated as another modality alongside video. UniPi, GR-1, DreamZero, LingBot-VA, VPP, mimic-video, and Fast-WAM all use some version of this. The risk is the modality mismatch: action chunks are different from the visual tokens the backbone was pretrained on, so the model has to adapt its representation during action fine-tuning.
Action as image
Another option is to turn the actions into something the video model already knows. Instead of new action tokens or a separate action head, encode actions as visual targets inside the same generation interface, so the pretrained video representation is not disrupted.
![Figure 17. GENIMA turns actions into visual targets: the image model predicts joint-action targets in RGB space, and a downstream controller maps those targets back to robot commands. Image from Shridhar et al., 2024 [31].](https://developer-blogs.nvidia.com/wp-content/uploads/2026/06/GENIMA-turns-actions-into-visual-targets.webp)
The closest early ancestor here is GENIMA [31]. GENIMA fine-tunes Stable Diffusion to draw joint-action targets on RGB images, then uses a controller to map those visual targets into joint-position actions. The interesting part is the interface choice: actions are expressed as something the generative image model can draw.
![Figure 18. Cosmos Policy latent injection: action, proprioception, and value targets are represented as synthetic latent frames inside the same video denoising interface. Image from Kim et al., 2026 [13].](https://developer-blogs.nvidia.com/wp-content/uploads/2026/06/Cosmos-Policy-latent-injection.webp)
A modern version of this direction is Cosmos Policy [13], which treats actions as synthetic latent video frames. Instead of adding a separate action decoder, it encodes action, proprioception, and value targets as fake frames inside the video model’s own denoising interface, and at inference time decodes the predicted action image back into an action vector by averaging spatial dimensions. This setup keeps the pretrained video backbone close to its native video denoising space while still producing robot actions.
Latent actions and plans
A different option is to compress behavior into latent plans or latent actions and condition the policy on those. This is attractive because full video prediction is expensive, and most pixels are not actually needed for control. Latent plans and latent actions are not exactly the same thing, but for this discussion I group them together: both are compact behavior abstractions learned from trajectories or video. The main differences are granularity and supervision. Plans usually cover a multi-step window and often need paired robot data; latent actions in the Genie/LAPA style can be learned from unlabeled video.
![Figure 19. Play-LMP architecture. A recognition network compresses trajectory windows into latent plans during training; at inference, a proposal network predicts the latent plan from the current observation and goal image. Image from Lynch et al., 2020 [32].](https://developer-blogs.nvidia.com/wp-content/uploads/2026/06/Play-LMP-architecture.webp)
Play-LMP [32] pioneered this idea in 2019. This is worth remembering because the basic idea is older than the current foundation-model wave. Play-LMP compressed sub-tasks into a small latent space as an intermediate abstraction for conditioning the low-level policy, long before access to today’s larger robot datasets and pretrained models. Concretely, a posterior network compressed short trajectory windows into a latent plan, a prior learned to predict that latent plan from the current observation and goal image, and a low-level policy decoded the sampled plan into actions.
The modern latent-action wave changes the scale and the data source. Genie [19] showed that latent action tokens can be learned from unlabeled internet video and used to drive an action-conditioned world model. Genie itself does not decode those latents into real robot motor commands, so it is not a robot policy. But it made the idea much more scalable: learn an action-like abstraction from video without having ground-truth robot actions. LAPA [33] then pushed this kind of latent-action pretraining toward VLA-style robot learning.
Being-H0.7 [42] is a modern WAM version of the original Play-LMP idea. It keeps the prior/posterior latent-plan logic, but executes it at foundation-model scale with several major changes. Instead of a small hierarchical latent-plan policy, it uses a larger Mixture-of-Transformers backbone. Similar to Play-LMP, the model has a posterior branch and a prior branch. The posterior branch gets access to future observations, encodes them with a frozen V-JEPA2.1 [64] visual encoder and Perceiver resampler, and compresses them into K future embeddings. The prior branch uses learnable latent queries and learns to match those future-informed latent states from the available context. At test time, the posterior branch is removed, so the policy gets a fast latent interface instead of forcing the model to regenerate full video sequences. The action generation part is still a flow-matching action policy. Being-H0.7 is trained on 200,000 hours of egocentric human video together with 15,000 hours of robot demonstrations.
![Figure 20. Being-H0.7 latent world-action architecture. The posterior branch compresses observed behavior into latent tokens, while the prior branch predicts those tokens for fast test-time policy inference. Image from BeingBeyond Team, 2026 [42].](https://developer-blogs.nvidia.com/wp-content/uploads/2026/06/Being-H0.7.webp)
| Design choice | Latent Plans / Play-LMP [32] | Being-H0.7 [42] |
|---|---|---|
| Main idea | Compress short robot behavior windows into latent plans that condition a low-level policy. | Learn a latent world-action model from large-scale egocentric video and robot demonstrations. |
| Data source | Robot play / demonstration trajectories. | 200k hours of egocentric human video plus 15k hours of robot demonstrations. |
| Architecture | Hierarchical latent-plan policy; LSTM low-level decoder. | Large MoT transformer for latent world-action modeling. |
| Latent variable | Trajectory-level latent plan, prior/posterior training. | Same prior/posterior structure, foundation-model scale. |
| Policy interface | Predict prior plan; low-level policy conditioned on observation and goal executes it. | Train both branches; at test time only the prior branch runs through the compact latent interface. |
The key difference is not the latent variable itself. Play-LMP already had the core prior/posterior latent-plan idea. Being-H0.7 shows how that interface can be scaled inside a modern WAM/VLA hybrid.
Latent actions have also become popular as an abstraction for action-conditioned world models. A recent example is DreamDojo [44], which learns continuous latent actions from large-scale egocentric human video for a controllable world model. The important distinction from inverse dynamics is the supervision path. Inverse-dynamics WAMs usually need paired video and action data to learn how visual transitions map to motor commands. Latent-action methods try to first learn a behavior abstraction from video itself, then connect that abstraction to robot action later.
Architecture: Hierarchical, Monolithic, or MoT?
The third axis is architecture: how the components are structurally composed. This is mostly orthogonal to the first two axes. Inverse dynamics can be hierarchical or MoT-style, joint prediction can be monolithic or expert-based, and latent-action methods can sit inside several different wrappers.
Hierarchical is the most flexible design because the action head is fully modular. It can be anything from a simple CNN regressor (UniPi) up to a full VLA stack (Pi-0.7’s BAGEL subgoals plus full VLA-based action expert), with VPP [24] and mimic-video [14] sitting in between by passing intermediate video-model features instead of full RGB rollouts. The downside is weak coupling between the video and action stages. Information flows one way, so this style is less natural when video and action should strongly influence each other.
Read the original source
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.