Method
An image and an instruction go into a frozen image-to-video diffusion model. At denoising step 16 of 40 we tap two signals out of it — a query–key attention track and pooled hidden states — and a small Reader turns them into a 6-DoF trajectory.
- The MOVE dataset. 5,038 egocentric object trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb–noun label.
- Trajectories from the latents, not from pixels. The generative backbone has already committed to how the object will move well before the video is rendered. DreamTraj decodes that commitment directly out of the model's intermediate representations, instead of finishing the generation and recovering motion from the rendered frames — to our knowledge the first method to do so.
- A 7.1M-parameter Reader. A lightweight flow-matching DiT decodes query–key attention tracks and pooled hidden states into 13 relative 6-DoF poses. The backbone stays frozen; the Reader is the only trained component.
MOVE dataset
5,038 egocentric object trajectories across six corpora, each paired with a fine-grained natural-language instruction rather than a coarse verb–noun label.
Interactive results
The scene is reconstructed from the single input frame; the object is placed at the 13 predicted poses. Drag to orbit, right-drag to pan, scroll to zoom — the camera is unconstrained, so it can leave the region the input image covers. Scrub the timeline to step through the trajectory, and press ⟲ to return to the original viewpoint.