K2V: Generative Keyframing For Physically Plausible Video Generation
arXiv:2611.17324
Felix O'Mahony, Daniel Olmeda Reino, Roberto Cipolla, Ayush Tewari
Free plain-English explainer
Video diffusion can paint convincing texture and still break basic physics, because the whole motion has to be invented inside one denoising process. K2V lets a language model set the important frames, then asks the video model only to fill the gaps.
Read free explainer →Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors. This is because, in a classical video generation setting, all physical reasoning must take place simultaneously and exclusively within the diffusion process.We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physically-correct intermediate states, rather than inferring an entire sequence with diffusion. Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the Physics-IQ Verified benchmark and PhyGenBench.