Diffusion models have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because
It has extra requirements on temporal consistency across frames, which naturally demand more world knowledge to be encoded into the model.

Lilian Weng is a machine learning researcher and writer who shares insights, research findings, and tutorials on machine learning, artificial intelligence, and data science. Through articles, blog posts, and research summaries, Lilian Weng explores topics such as deep learning, natural language processing, and reinforcement learning. Readers can learn about state-of-the-art algorithms, practical applications of machine learning, and trends shaping the field of AI.

Lil’Log

Diffusion models have been successful in image synthesis and now researchers are exploring their application in video generation. Video generation is more challenging than image generation due to additional requirements on temporal consistency and the scarcity of high-quality video data. Different approaches for video generation modeling from scratch and adaptation from pre-trained image models are discussed.

Diffusion Models for Video Generation

Adapting Image Models to Generate Videos #