ComfyUI Extension: ComfyUI-Advanced-ControlNet

ComfyUI-Advanced-ControlNet Introduction

ComfyUI-Advanced-ControlNet is an extension designed to enhance the capabilities of ControlNet within the ComfyUI framework. This extension introduces advanced features such as scheduling ControlNet strength across different timesteps and batched latents, applying custom weights, and utilizing attention masks. These features allow AI artists to have more granular control over their image generation processes, enabling the creation of more complex and refined outputs.

The extension supports various ControlNet types, including T2IAdapters, ControlLoRAs, ControlLLLite, SparseCtrls, SVD-ControlNets, and Reference. By using this extension, AI artists can replicate features from other popular tools, such as the "My prompt is more important" and "ControlNet is more important" features from Auto1111's sd-webui ControlNet extension, but with more customization options.

How ComfyUI-Advanced-ControlNet Works

At its core, ComfyUI-Advanced-ControlNet works by allowing users to schedule and control the influence of ControlNet on their image generation process. This is achieved through two main concepts: Timestep Keyframes and Latent Keyframes.

• Timestep Keyframes: These keyframes hold values that guide the settings for ControlNet. They take effect based on their start_percent, which corresponds to a percentage of the sampling process. Timestep Keyframes can include masks for strengths, control_net_weights, and latent_keyframes.
• Latent Keyframes: These keyframes determine the strength of ControlNet for specific latents. They contain the batch index of the latent and the strength that should be applied. This allows for precise control over how much influence ControlNet has on different parts of the image. By using these keyframes, AI artists can create complex schedules that dictate how ControlNet should behave throughout the image generation process, leading to more dynamic and controlled outputs.

ComfyUI-Advanced-ControlNet Features

Timestep and Latent Strength Scheduling

This feature allows you to schedule the strength of ControlNet across different timesteps and latents. You can define keyframes that specify when and how ControlNet should be applied, providing fine-grained control over the image generation process.

Attention Masks

Attention masks let you decide which parts of the image ControlNet should focus on. By applying these masks, you can control the relative strength of ControlNet's influence on different areas of the image.

Custom Weights

Custom weights enable you to replicate features from other tools, such as the "My prompt is more important" and "ControlNet is more important" features from Auto1111's sd-webui ControlNet extension. You can adjust the softness and importance of these weights to achieve the desired effect.

Sliding Context Windows

This feature supports sliding context sampling, similar to the one used in the ComfyUI-AnimateDiff-Evolved nodes. It allows for more stable and coherent outputs by maintaining context across different parts of the image.

Support for Various ControlNet Types

The extension supports multiple ControlNet types, including:

• ControlNets
• T2IAdapters
• ControlLoRAs
• ControlLLLite
• SparseCtrls
• SVD-ControlNets
• Reference Each type offers different capabilities and can be used to achieve various effects in your image generation process.

ComfyUI-Advanced-ControlNet Models

The extension supports several models, each designed for specific use cases:

• Stable Video Diffusion ControlNets: Trained by CiaraRowles, these models include Depth and Lineart.
• Reference ControlNet: Supports reference_attn, reference_adain, and reference_adain+attn modes. It allows for fine-tuning style fidelity, weight, and strength of attention and adain separately.

Troubleshooting ComfyUI-Advanced-ControlNet

Common Issues and Solutions

1. ControlNet Not Taking Effect:

• Ensure that the start_percent and stop_percent values are correctly set.
• Check if the appropriate keyframes are connected and configured.

2. Unexpected Output:

• Verify the custom weights and attention masks to ensure they are set as intended.
• Review the scheduling of keyframes to ensure they align with your desired output.

3. Errors with ControlNet Types:

• Make sure the ControlNet type you are using is supported by the extension.
• Check for compatibility issues with other nodes or extensions.

Frequently Asked Questions

• How do I apply custom weights?
• Custom weights can be applied through the weights_override input on the Apply Advanced ControlNet node. Adjust the base_multiplier and uncond_multiplier to achieve the desired effect.
• What is the difference between Timestep and Latent Keyframes?
• Timestep Keyframes control the overall settings for ControlNet at specific points in the sampling process, while Latent Keyframes determine the strength of ControlNet for specific latents.

Learn More about ComfyUI-Advanced-ControlNet

For additional resources, tutorials, and community support, you can explore the following links:

• ComfyUI-AnimateDiff-Evolved
• ComfyUI ControlNet Auxiliary Preprocessors
• FizzNodes Wiki These resources provide valuable information and examples to help you get the most out of ComfyUI-Advanced-ControlNet.