ComfyUI Extension: AnimateDiff

AnimateDiff Introduction

The comfyui-animatediff extension integrates the powerful AnimateDiff technology into ComfyUI, allowing AI artists to create stunning animations from text prompts or images. This extension adapts from the sd-webui-animatediff and provides a seamless way to generate animated content without needing extensive technical knowledge. With comfyui-animatediff, you can transform your creative ideas into dynamic animations, making it an invaluable tool for digital artists, animators, and content creators.

How AnimateDiff Works

At its core, comfyui-animatediff leverages motion modules to inject movement into static images generated by AI models. Think of it as adding a layer of animation on top of your AI-generated images. The process involves several key steps:

1. Loading Motion Modules: These modules contain pre-trained data that guide the animation process.
2. Sampling Frames: The extension generates multiple frames based on the input image or text prompt.
3. Combining Frames: These frames are then combined to create a smooth animation, which can be saved in various formats like GIF, WebP, or MP4. By breaking down the animation process into these steps, comfyui-animatediff simplifies the creation of complex animations, making it accessible even to those with minimal technical background.

AnimateDiff Features

AnimateDiffLoader

This node loads the motion module required for generating animations. It is the first step in the animation workflow.

AnimateDiffSampler

Similar to the KSampler, this node handles the sampling of frames. Key settings include:

• motion_module: Specifies the motion module to use.
• frame_number: Determines the length of the animation.
• latent_image: Allows passing an EmptyLatentImage for sampling.
• sliding_window_opts: Customizes sliding window options for generating longer animations.

AnimateDiffCombine

This node combines the sampled frames into a final animation. Key settings include:

• frame_rate: Sets the number of frames per second.
• loop_count: Determines how many times the animation should loop (use 0 for infinite loop).
• save_image: Decides whether to save the animation to disk.
• format: Supports various formats like GIF, WebP, WebM, and MP4.

SlidingWindowOptions

Customizes the sliding window feature, which is useful for generating long animations without frame length limits. Key settings include:

• context_length: Number of frames per window.
• context_stride: Sampling strategy for frames.
• context_overlap: Overlap between window slices.
• closed_loop: Creates a closed-loop animation.

LoadVideo

This node allows loading existing GIFs or videos as input images, which can be useful for using them as ControlNet inputs.

AnimateDiff Models

The extension supports various models, each suited for different types of animations:

• Original Motion Modules: Available on Google Drive, HuggingFace, CivitAI, and Baidu NetDisk.
• Community Modules: Includes AD_Stabilized_Motion and TemporalDiff.
• AnimateDiff v2: The latest version, available here.

What's New with AnimateDiff

Update 2023/09/25

• Motion LoRA Support: Download motion LoRAs and place them in the comfyui-animatediff/loras/ folder. Note that LoRAs only work with AnimateDiff v2. - New Node: AnimateDiffLoraLoader: This node loads the motion LoRAs for use in animations.

Update 2023/09/21

• Sliding Window Feature: Automatically activated when generating more than 16 frames, this feature divides frames into smaller batches with slight overlap, enabling the creation of longer animations.

Troubleshooting AnimateDiff

Common Issues and Solutions

CUDA Error: Invalid Configuration Argument

This is a known issue with xformers. The current workaround is to disable xformers by adding --disable-xformers when starting ComfyUI.

GIF Split into Multiple Scenes

If your GIF is splitting into multiple scenes, try the following:

• Shorten your prompt and negative prompt.
• Reduce the resolution to 512x512, as AnimateDiff is trained on this resolution.
• Disable xformers with --disable-xformers.

GIF Has Watermark

This issue is due to the training data containing Shutterstock watermarks. Try using other community-finetuned modules to avoid this problem.

Learn More about AnimateDiff

For additional resources, tutorials, and community support, consider the following:

• ComfyUI Documentation
• AnimateDiff Project Page
• Hugging Face Spaces By exploring these resources, you can deepen your understanding of comfyui-animatediff and enhance your animation projects.