ComfyUI Extension: ComfyUI-FLATTEN

ComfyUI-FLATTEN Introduction

ComfyUI-FLATTEN is an extension for ComfyUI that integrates the FLATTEN model, which is designed to enhance the consistency of text-to-video editing. This extension provides a set of nodes that leverage the FLATTEN model's capabilities to ensure that all frames in an edited video remain visually consistent. This is particularly useful for AI artists who want to create seamless and coherent video edits based on textual prompts.

The FLATTEN model introduces optical flow into the attention mechanism of the diffusion model's U-Net, which helps maintain visual consistency across frames. This extension can be a powerful tool for artists looking to create high-quality video edits without the need for extensive training or complex setups.

How ComfyUI-FLATTEN Works

At its core, ComfyUI-FLATTEN uses the FLATTEN model to process video frames by incorporating optical flow into the attention mechanism. Optical flow is a technique used to track the movement of objects between consecutive frames in a video. By integrating this into the attention mechanism, FLATTEN ensures that corresponding patches across different frames attend to each other, thereby maintaining consistency.

Think of it like a painter who ensures that the colors and strokes used in one part of a mural match those used in adjacent parts, even as the mural progresses. This way, the entire mural looks cohesive and well-integrated. Similarly, FLATTEN ensures that the visual elements in each frame of a video are consistent with those in the surrounding frames.

ComfyUI-FLATTEN Features

Load Checkpoint with FLATTEN Model

• Description: This node loads any given Stable Diffusion 1.5 (SD1.5) checkpoint with the FLATTEN optical flow model.
• Customization: Use the sdxl branch to load SDXL models.
• Usage: Ideal for loading models that will be used with the Flatten KSampler.

Sample Trajectories

• Description: Samples the optical flow of input images into trajectories.
• Customization:
• Context Length: Defines the window size FLATTEN processes at a time.
• Context Overlap: Defines the overlap between windows.
• Usage: Useful for reducing VRAM usage and ensuring consistency in video frames.

Unsampler (Flatten)

• Description: Unsamples the input latent and creates the necessary injections for sampling.
• Customization: Use Euler or ddpm2m as the sampling method.
• Usage: Best for creating noise from input images.

KSampler (Flatten)

• Description: Samples the unsampled latents using injections from the Unsampler.
• Customization: Can use any sampling method, but Euler or ddpm2m are recommended for video editing.
• Usage: Ideal for editing specific parts of a video or making significant changes.

Apply Flatten Attention (SD1.5 Only)

• Description: Uses FLATTEN's optical flow attention mechanism without the rest of the model.
• Customization: Can be combined with other models.
• Usage: Useful for adding flow noise to enhance consistency.

Create Flow Noise

• Description: Generates flow noise given a latent and trajectories.
• Usage: Adds initial noise to a latent instead of using traditional noise from a KSampler.

ComfyUI-FLATTEN Models

ComfyUI-FLATTEN supports different models, each suited for specific tasks:

• SD1.5 Checkpoints: Use these for standard video editing tasks.
• SDXL Models: Available on the sdxl branch for more advanced editing needs.

When to Use Each Model

• SD1.5 Checkpoints: Best for general use and compatibility with most features.
• SDXL Models: Use when higher resolution or more complex edits are required.

Troubleshooting ComfyUI-FLATTEN

Common Issues and Solutions

1. Model Not Loading:

• Solution: Ensure you are using the correct branch (sdxl for SDXL models) and that the model files are in the correct directory.

2. Inconsistent Frames:

• Solution: Adjust the context length and overlap settings to ensure better consistency.

3. High VRAM Usage:

• Solution: Reduce the context length or use batching options to manage VRAM usage.

Frequently Asked Questions

• Q: Can I use other sampling methods with the KSampler (Flatten)?
• A: Yes, but Euler or ddpm2m are recommended for the best results.
• Q: How do I add flow noise?
• A: Use the Create Flow Noise node to generate and add flow noise to your latents.

Learn More about ComfyUI-FLATTEN

For additional resources, tutorials, and community support, consider the following:

• Original Research Repo: FLATTEN
• Project Website: FLATTEN Project Website
• Video Helper Suite: Video Helper Suite
• AnimateDiff-Evolved: AnimateDiff-Evolved These resources provide comprehensive information and examples to help you get the most out of ComfyUI-FLATTEN.