ComfyUI Extension: BrushNet

BrushNet Introduction

ComfyUI-BrushNet is an extension for ComfyUI that integrates advanced image inpainting models to enhance your creative workflows. This extension includes custom nodes for the following models:

• BrushNet: A plug-and-play image inpainting model that uses decomposed dual-branch diffusion to achieve high-quality results.
• PowerPaint: A versatile image inpainting model that leverages task prompts for high-quality outputs.
• HiDiffusion: A model designed to unlock higher-resolution creativity and efficiency in pretrained diffusion models. These models can help AI artists by providing powerful tools for image inpainting, object removal, outpainting, and more, making it easier to create and refine digital artworks.

How BrushNet Works

ComfyUI-BrushNet works by integrating advanced inpainting models into the ComfyUI framework. These models use diffusion processes to fill in missing parts of an image or modify existing parts based on user inputs. The extension provides custom nodes that allow you to load and apply these models within your ComfyUI workflows.

Basic Principles

1. Diffusion Models: These models generate images by iteratively refining random noise into coherent images based on a given prompt or conditioning input.
2. Inpainting: This technique involves filling in missing or masked parts of an image. The models use the surrounding context to generate plausible content for the masked areas.
3. Task Prompts: PowerPaint uses task-specific prompts to guide the inpainting process, allowing for more control over the final output.

BrushNet Features

BrushNet

• Scale: Adjusts the strength of the BrushNet model's influence on the image.
• Start/End Steps: Defines the steps at which the BrushNet model starts and stops applying its effects.

PowerPaint

• CLIP: Uses a CLIP model to guide the inpainting process.
• Fitting Degree: Controls how closely the inpainting follows the task prompt.
• Function: Specifies the type of inpainting task (e.g., object removal, outpainting).
• Save Memory: Reduces memory usage at the cost of speed.

RAUNet

• Downsample/Upsample Resize: Controls the steps at which resizing starts and stops.
• CrossAttention Resize: Manages the steps for cross-attention resizing.

BrushNet Models

BrushNet Models

• segmentation_mask_brushnet_ckpt: Trained on BrushData with segmentation masks.
• random_mask_brushnet_ckpt: General model for random mask shapes.
• segmentation_mask_brushnet_ckpt_sdxl_v0: SDXL version with segmentation masks.
• random_mask_brushnet_ckpt_sdxl_v0: SDXL version for random mask shapes.

PowerPaint Models

• PowerPaint v2: Enhanced model for versatile inpainting tasks.

What's New with BrushNet

Recent Updates

• May 16, 2024: Internal rework for better compatibility. RAUNet implemented.
• May 12, 2024: CutForInpaint node added.
• May 11, 2024: Image batch processing implemented.
• May 6, 2024: PowerPaint v2 model added.
• May 2, 2024: BrushNet SDXL version released.
• Apr 28, 2024: Complete rework for native ComfyUI integration.

Troubleshooting BrushNet

Common Issues and Solutions

1. Workflow Not Working After Update:

• Check the end_at parameter of the BrushNode. If it equals 1, change it to a larger number.

2. Compatibility Issues:

• Some nodes may not be compatible with BrushNet due to overlapping functionalities. Known incompatible nodes include:
• WASasquatch's FreeU_Advanced
• blepping's jank HiDiffusion

3. Out of Memory (OOM) Errors:

• Use the Evolved Sampling feature from AnimateDiff-Evolved to process images in chunks.

Frequently Asked Questions

1. Why is my inpainted image blurry?

• This can happen due to inference and VAE issues. Use the blending inpaint workflow to repair the image.

2. How do I remove an object completely?

• Add object descriptions to the negative prompt and describe an empty scene.

Learn More about BrushNet

Additional Resources

• BrushNet Paper
• PowerPaint Paper
• HiDiffusion Paper
• ComfyUI Documentation
• BrushNet Project Page
• PowerPaint Project Page Explore these resources to deepen your understanding and get the most out of ComfyUI-BrushNet.