ComfyUI Extension: ComfyUI-AnimateAnyone-Evolved

ComfyUI-AnimateAnyone-Evolved Introduction

ComfyUI-AnimateAnyone-Evolved is an advanced extension designed to transform static images into dynamic, stylized videos. This tool leverages the power of AI to animate characters and scenes based on pose image sequences and reference images. It is particularly useful for AI artists looking to bring their creations to life with minimal effort. The extension aims to achieve a frame rate of 1+ FPS on GPUs equivalent to or better than the RTX 3080, making it accessible for users with high-end hardware.

How ComfyUI-AnimateAnyone-Evolved Works

ComfyUI-AnimateAnyone-Evolved works by taking a sequence of pose images and a reference image to generate a video that animates the reference image according to the poses. Think of it as a digital puppeteer: the pose images act as the strings that control the movements, while the reference image is the puppet. The extension uses various AI models and algorithms to ensure that the animation is smooth and visually appealing.

Basic Principles:

1. Pose Image Sequence: A series of images that depict different poses or movements.
2. Reference Image: The static image that you want to animate.
3. AI Models: These models process the pose image sequence and the reference image to generate the animated video.

ComfyUI-AnimateAnyone-Evolved Features

Sampler & Scheduler Support

The extension supports various samplers and schedulers, each offering different performance and quality trade-offs:

• DDIM:
• 24 frames, steps=20, context_frames=24; Takes 835.67 seconds on RTX 3080.

• 24 frames, steps=20, context_frames=12; Takes 425.65 seconds on RTX 3080.

• DPM++ 2M Karras:
• 24 frames, steps=20, context_frames=12; Takes 407.48 seconds on RTX 3080.

• LCM:
• 24 frames, steps=20, context_frames=24; Takes 606.56 seconds on RTX 3080.

• Euler:
• 24 frames, steps=20, context_frames=12; Takes 450.66 seconds on RTX 3080.

• Euler Ancestral
• LMS
• PNDM

Lora Support

You can add Lora models to enhance the animation quality. This feature allows for more detailed and customized animations.

Long Pose Image Sequences

The extension can handle long pose image sequences, tested up to 120+ frames on an RTX 3080 GPU. The main parameter affecting GPU usage is context_frames, which does not correlate with the length of the pose image sequences.

Modular Workflow

The implementation is broken down into modules, making the workflow in ComfyUI closely resemble the original pipeline from the AnimateAnyone paper.

AA_pipeline

ComfyUI-AnimateAnyone-Evolved Models

The extension supports various models, each suited for different tasks and performance levels:

• Stable-Diffusion-v1-5_unet: General-purpose model for high-quality animations.
• Moore-AnimateAnyone Pre-trained Models: Specialized models for specific animation tasks. These models need to be placed in the pretrained_weights folder as follows:

./pretrained_weights/
|-- denoising_unet.pth
|-- motion_module.pth
|-- pose_guider.pth
|-- reference_unet.pth
`-- stable-diffusion-v1-5
  |-- feature_extractor
  |  `-- preprocessor_config.json
  |-- model_index.json
  |-- unet
  |  |-- config.json
  |  `-- diffusion_pytorch_model.bin
  `-- v1-inference.yaml

What's New with ComfyUI-AnimateAnyone-Evolved

Recent Updates

• Residual CFG from StreamDiffusion: Implemented components to speed up the process by about 2X.
• New Models: Plans to incorporate models from Open-AnimateAnyone and AnimateAnyone once released.
• Speed Improvements: Ongoing efforts to convert models using stable-fast for an estimated 2X speedup.
• Quality Enhancements: Training new models with better datasets to improve result quality.

Troubleshooting ComfyUI-AnimateAnyone-Evolved

Common Issues and Solutions

1. Slow Performance: Ensure your GPU meets the minimum requirements (RTX 3080 or better). Try reducing the context_frames parameter.
2. Model Loading Errors: Verify that all models are correctly placed in the pretrained_weights folder.
3. Artifacts in Animation: Experiment with different samplers and schedulers to find the best settings for your specific use case.

FAQs

• Can I use this on a lower-end GPU?
• The extension is optimized for high-end GPUs like the RTX 3080. Performance on lower-end GPUs may not be satisfactory.
• How do I add a new Lora model?
• Place the Lora model in the appropriate directory and update the configuration to include it.

Learn More about ComfyUI-AnimateAnyone-Evolved

Additional Resources

• Example Workflows: Check the example workflows for detailed usage instructions.
• Community Forums: Join AI art communities and forums to share your experiences and get support.
• Tutorials and Documentation: Look for tutorials and detailed documentation to get the most out of this extension. By following these guidelines, you can create stunning animations with ComfyUI-AnimateAnyone-Evolved, bringing your AI-generated art to life in new and exciting ways.