Can 4D Foundation Models Remember?

The 30-second take

• What: PersistBench scores 4D foundation models on object permanence, motion continuity, and appearance preservation, using 360° videos so objects that leave one view still have a known reference.
• Abundance angle: today, robust visual memory — knowing what left the frame and how it should look when it returns — is still an elite research capability, not a default in robots or AR. Measuring that gap is a step toward memory-grade 4D tools as ordinary infrastructure (near-term for benchmarks; longer for products).
• Who should care: Computer-vision and robotics groups building camera-controllable video or 4D reconstruction models, and product teams who assume 'the model saw it' means 'the model still knows it.'

What the paper actually did

The authors ask whether current 4D foundation models — camera-controllable video models and 4D reconstruction models — remember what they have perceived, not only whether they can reconstruct a dynamic scene. Existing benchmarks, they argue, lean on pixel-level metrics and lack ground truth for objects after those objects leave the field of view, so they cannot score visual memory in an object-centric way against references.

They introduce PersistBench: a dataset and metric suite that treats 360° videos as omniscient ground truth. Evaluation has three axes: object permanence, motion continuity, and appearance preservation. They run a range of models across categories and report that current systems keep only short-term consistency, which degrades sharply once objects leave the field of view.

The headline finding is framed as “seeing is not remembering.” Dataset and code are pointed to https://guangzhaohe.com/persistbench.

What makes this disruptive

The scarce capability is persistent, object-centric memory in models that already look like they understand 4D scenes. If PersistBench becomes a standard, papers that only win on pixels while the camera is pointed at an object will look incomplete.

That pressures a luxury: systems that seem spatially fluent in demos but drop objects the moment they exit frame. A public 360° omniscient protocol makes that failure measurable instead of anecdotal.

This is a diagnostic paper, not a new foundation model. Its disruptiveness is in changing what “good 4D” means — a roadmap signal for the next training objective, not a claim that memory is solved.

Why it matters (outside the lab)

Abundance lens: cheap, default spatial intelligence for robots, vehicles, and AR needs memory, not just pretty reconstruction. If models forget occluded or off-screen objects, every downstream planner must rebuild state from scratch — keeping scene understanding scarce and brittle.

Near-term, PersistBench is a yardstick. Medium-term, whether labs train for permanence, continuity, and appearance — and whether those skills transfer — decides if 4D tools become ordinary infrastructure.

No invented product date. The paper’s own moral is that perception without memory is not enough.

Limitations & open questions

A benchmark does not fix the models. The abstract does not list which architectures were evaluated or numeric scores, only a qualitative pattern: short-term consistency that collapses after objects leave view. 360° video as “omniscient” ground truth still depends on capture quality, annotation protocol, and how objects are defined — details not in the abstract.

Pixel-level metrics are criticized but not replaced for all use cases; some applications still care about image fidelity. Preprint ≠ deployed memory module. Abundance is not automatic: a harder test does not by itself make visual memory a consumer default.