TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

The 30-second take

• What: TrackEverything represents video as persistent 3D scene tracks, de-duplicates overlapping sliding-window tracks with voxelization, refines endpoints then only dynamic trajectories, and replaces 4D correlation volumes with 3D feature sampling — reporting >20% APD over open-source dense 3D trackers on short TAPVid-3D clips.
• Abundance angle: today, long, dense 3D tracking is an elite-GPU luxury. Memory that scales with unique scene geometry rather than video length is a step toward default spatial memory for robots and reconstruction (mid-horizon: reliability and hardware still decide).
• Who should care: 3D vision and robotics teams who need all-point tracks, and anyone who has had to choose sparse-and-long versus dense-and-short.

What the paper actually did

Existing point trackers, the authors say, face a trade-off: track a sparse set of queries for a long time, or track all points on short clips. TrackEverything is a 3D point tracker that treats video as persistent 3D scene tracks in world coordinates, on the idea that video is a 2D projection of a 3D world. That is meant to decouple model complexity from video duration so cost scales with unique physical geometry instead.

Three pieces: a voxelization-based de-duplication at sliding-window boundaries that merges co-located tracks so the same surface is not stored again; a split of tracking into an endpoint refiner (destination plus static-versus-dynamic classification) and a lightweight trajectory refiner that decodes dense paths only for dynamic points; and 3D WAFT, which replaces memory-heavy 4D correlation volumes with feature sampling in the scene cloud.

They say it is the first 3D tracker that can follow all visible points on videos longer than 1000 frames within 40 GB of GPU memory. On TAPVid-3D it outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while staying competitive with state-of-the-art sparse trackers on long sequences despite tracking far more points.

What makes this disruptive

The scarce capability is dense, long-horizon 3D correspondence — the memory wall that keeps “track everything” a short-clip demo. If complexity follows unique surfaces, minute-scale dense tracks become thinkable on a single 40 GB card.

Voxel de-dup plus static/dynamic split plus 3D (not 4D) correlation is a clear systems bet. The >20% APD gap versus open-source dense 3D trackers on TAPVid-3D short clips is the scoreboard line; competitiveness with sparse SOTA on long video is the other.

“First” and memory claims are the authors’. Treat them as a methods milestone, not a product SLAM stack.

Why it matters (outside the lab)

Abundance lens: persistent 3D memory of a scene is still luxury compute. If dense tracks fit in ordinary high-end GPUs, more robots, AR, and reconstruction pipelines can treat long spatial memory as a default rather than a research clip.

Near-term, this is a tracker baseline and a memory architecture. Medium-term, dynamic scenes, independent reimplementation, and whether 40 GB stays the bar decide if it becomes ordinary.

No date. Better tracking does not by itself make household robots.

Limitations & open questions

Preprint; we have not measured the 40 GB / 1000-frame claim. TAPVid-3D short-clip APD gains are versus open-source dense 3D trackers they include, not necessarily every unpublished system. “Competitive” with sparse SOTA on long sequences is qualitative in the abstract.

The abstract does not report speed, accuracy on heavy occlusion, or how voxel size affects merge errors. Static/dynamic classification mistakes would skip trajectory refinement. World-coordinate tracks assume a consistent 3D frame.

Abundance is not automatic: a memory-efficient tracker does not demonetize 3D perception hardware.