6-DoF Pose Tracking

ProxyPose: 6-DoF Pose Tracking
via Video‑to‑Video Translation

Ruihang Zhang^*1, Felix Taubner^*1,2, Pooja Ravi¹, Kiriakos N. Kutulakos^1,2, David B. Lindell^1,2

1 University of Toronto · 2 Vector Institute · * Equal contribution

TL;DR — one query pixel in, a full 6‑DoF pose trajectory out.

Overview

One point in, one trajectory out

Given a source video and a single query pixel, ProxyPose first translates the input into a "proxy video" in which a colored cube undergoes the same local rigid-body motion as the queried point, then recovers the 6-DoF pose trajectory via Perspective-n-Point (PnP).

monocular video + query pixel → proxy video → 6-DoF pose trajectory

Gallery

Browse tracked sequences

Click to switch between 3D tracking visualizations and the generated proxy videos. Select sequences using the thumbnails below.

querytracking3D visualization

Method

Video-to-video translation, then classical geometry

Overview of ProxyPose. (a) The method takes as input a monocular RGB video and a user-selected query point, both encoded into the latent space of a variational autoencoder. During training, first-frame proxy tokens receive reduced noise for conditioning, subsequent proxy tokens are corrupted with full Gaussian noise, and source video tokens remain noise-free. The source and proxy token streams are concatenated along the token dimension and processed by a LoRA-finetuned video generative model to synthesize a temporally consistent proxy video. (b) The generated proxy video is tracked using learning-free geometric methods—contour fitting and Perspective-n-Point (PnP) optimization—to recover the 6-DoF pose of the queried object.

Applications

Beyond object pose

Camera tracking

Our method can be extended to camera pose tracking by tracking a static point in the world. This enables robust tracking even when COLMAP fails, for example, due to lack of texture.

Face tracking

Despite being trained only on rigid objects, ProxyPose can also track non-rigid surfaces such as human faces.

ProxyPose tracking FlowFace tracking

Extension to other sensing modalities

With no additional training, ProxyPose can be applied to other sensing modalities such as event-based video and single-photon imaging. Here we track a hand spinner captured with an event camera and a Nerf gun captured with a single-photon avalanche diode (SPAD) array.

Event-based video Single-photon imaging

Baseline comparisons

In-the-wild videos and rigid pose datasets

We conduct experiments on 6-DoF pose tracking across various datasets. For quantitative results, we refer the reader to our paper.

In the wild videos

Previous methods struggle to track objects with challenging materials (e.g., transparent, reflective) and under extreme occlusions. ProxyPose handles these challenging scenarios robustly.

ProxyPoseCoTracker+DAV3BundleSDF

ProxyPoseCoTracker+DAV3

Rigid Pose Tracking Datasets

We evaluate on rigid pose estimation datasets HO3D, YCBInEOAT, and a set of sequences from the ProxyPose Synthetic Benchmark. Since the objects in these datasets are rigid, multi-query bundle adjustment can be applied, which improves performance in most cases. However, even without incorporating rigidity constraints or bundle adjustment, ProxyPose (one query) achieves state-of-the-art performance.

Limitations

Failure Cases

Despite its generality, ProxyPose inherits limitations from the underlying video model and proxy-based formulation. We sometimes observe pose drift for reflective or textureless surfaces under complex motion (balloons). Additionally, the approximate local rigidity implied by the proxy can be inconsistent with scenes where tracking is ill-defined (e.g., surface regions on a fluid). Fast motion can exceed the capabilities of the video model's variational autoencoder, producing blurred proxy frames that degrade contour detection and tracking (see the marble racing scene).

Reflective surfaces (balloons)

Ill-defined motion (ocean waves)

Fast motion (marble racing)

Citation

BibTeX

@article{zhang2026proxypose,
  title={ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation},
  author={Ruihang Zhang and Felix Taubner and Pooja Ravi and Kiriakos N. Kutulakos and David B. Lindell},
  journal={arXiv preprint arXiv:2607.06555},
  year={2026}
}