ComfyUI Extension: ComfyUI_StoryDiffusion

ComfyUI_StoryDiffusion Introduction

ComfyUI_StoryDiffusion is an extension designed to enhance the capabilities of ComfyUI by integrating the StoryDiffusion framework. This extension allows AI artists to generate consistent and coherent images and videos from textual prompts, making it easier to create visual stories and animations. By leveraging advanced self-attention mechanisms and motion prediction models, ComfyUI_StoryDiffusion ensures that characters and scenes remain consistent across long sequences, solving common issues of inconsistency in multi-frame image generation.

How ComfyUI_StoryDiffusion Works

ComfyUI_StoryDiffusion operates on the principles of consistent self-attention and motion prediction. Here's a simplified breakdown:

1. Consistent Self-Attention: This mechanism ensures that characters and elements in the images remain consistent across multiple frames. It works by maintaining a coherent representation of characters and scenes, even when generating long sequences of images.
2. Motion Prediction: For video generation, the motion predictor model predicts the movement between frames in a compressed semantic space. This allows for smooth transitions and larger motion predictions, resulting in high-quality, long-range videos. By combining these two techniques, ComfyUI_StoryDiffusion can generate both static images and dynamic videos that are visually coherent and narratively consistent.

ComfyUI_StoryDiffusion Features

Dual Role Same Frame Function

• Usage: To place two characters in the same frame, use the format (A and B) have lunch..., where A and B are the character names. The "and" and parentheses must remain for the function to work.
• Customization: Adjust parameters like role_scale, mask_threshold, and start_step to control the randomness and style consistency of the characters.

Lora Integration

• Optimized Loading: When using accelerated Lora, trigger_words are no longer added to the prompt list, improving performance.
• Customization: Adjust ip_adapter_strength and style_strength_ratio in img2img mode to fine-tune style consistency.

Playground v2.5

• Functionality: Effective in txt2img mode, allowing for the use of style Lora when accelerated Lora is available.

Model Loading

• Separation: The model loading node is separated to handle the numerous adjustable parameters efficiently.
• Support for SDXL Models: You can include your favorite SDXL-based diffusion models by editing the config/models.yaml file.

Preprocess Translation Text Nodes

• Usage: Use the example diagram for guidance. Ensure to change the font for Chinese or other East Asian characters.

ComfyUI_StoryDiffusion Models

Required Models

• Encoder Model: laion/CLIP-ViT-bigG-14-laion2B-39B-b160k (cannot be replaced).
• IP Adapter Fine-Tuning Model: ms_adapter.bin (cannot be replaced).

Optional Models

• Photomaker Model: photomaker-v1.bin for image generation processes.

What's New with ComfyUI_StoryDiffusion

Latest Updates

1. Dual Role Same Frame: Added functionality to place two characters in the same frame.
2. Model Requirements: Integrated MS-Diffusion features, requiring specific encoder and adapter models.
3. Lora Code Optimization: Improved loading and performance of Lora models.
4. Playground v2.5: Enhanced functionality in txt2img mode.
5. Parameter Management: Separated model loading node and removed unnecessary three-role nodes.
6. Style Consistency: New parameters to adjust style consistency and randomness in dual role frames.
7. Img2Img Adjustments: Fine-tune style consistency with ip_adapter_strength and style_strength_ratio.

Troubleshooting ComfyUI_StoryDiffusion

Common Issues and Solutions

1. Model Loading Errors: Ensure all required models are correctly placed in the specified directories. If using the integrated package, follow the specific installation steps provided.
2. Dual Role Function Not Working: Verify that the format (A and B) is used correctly and that the required models are loaded.
3. Lora Integration Issues: Check that trigger_words are correctly set and that the Lora model is compatible.

Frequently Asked Questions

• Q: How do I add new models?
• A: Edit the config/models.yaml file and follow the same format to include new SDXL-based diffusion models.
• Q: What if the image generation is inconsistent?
• A: Adjust parameters like role_scale, mask_threshold, and start_step to improve consistency.

Learn More about ComfyUI_StoryDiffusion

Additional Resources

• StoryDiffusion Project Page: StoryDiffusion
• MS-Diffusion Repository: MS-Diffusion
• IP-Adapter Repository: IP-Adapter

Community and Support

• ComfyUI Community Forums: Engage with other AI artists and developers to share tips, ask questions, and get support.
• Tutorials and Documentation: Explore detailed tutorials and documentation to make the most out of ComfyUI_StoryDiffusion. By following this comprehensive guide, you can effectively utilize ComfyUI_StoryDiffusion to create stunning and consistent visual stories and animations.