ComfyUI Extension: ComfyUI fabric

ComfyUI fabric Introduction

ComfyUI_fabric is an extension for ComfyUI that integrates nodes based on the research paper "FABRIC: Personalizing Diffusion Models with Iterative Feedback." This extension allows AI artists to personalize their diffusion models by incorporating iterative feedback through attention-based reference image conditioning. Essentially, it enables you to guide the generative process by providing positive and negative reference images, making it easier to achieve desired artistic outcomes without extensive prompt tweaking.

Read the Paper | Main Repository | SD Web UI Extension

How ComfyUI fabric Works

ComfyUI_fabric leverages the self-attention mechanism in the U-Net architecture of diffusion models. By conditioning the diffusion process on a set of positive and negative reference images, the extension allows you to iteratively refine the generated images. Think of it as a way to "teach" the model what you like and dislike, guiding it to produce images that align more closely with your artistic vision.

For example, if you provide images of styles you like (positive feedback) and styles you dislike (negative feedback), the model will adjust its output to incorporate these preferences. This iterative feedback loop continues until the generated images meet your expectations.

ComfyUI fabric Features

Nodes

ComfyUI_fabric introduces several nodes that you can find under the FABRIC/ category in the node menu:

• FABRIC Patch Model: Patches a model to use FABRIC, making it compatible with any sampler node.
• FABRIC Patch Model (Advanced): Similar to the basic model patcher but with additional inputs for null_pos and null_neg instead of a clip input.
• KSampler FABRIC: Functions like a KSampler but includes full FABRIC inputs.
• KSampler FABRIC (Advanced): An advanced version of KSampler with full FABRIC inputs.
• KSampler FABRIC (Simple): A simplified version of KSampler with intended FABRIC inputs.
• Batch Latents: A helper node for combining two latents in a batch, useful for using multiple positive/negative latents in FABRIC.

Parameters

• pos_latents: Optional input for latents you want the resulting latent to borrow characteristics from (e.g., "liked" images).
• neg_latents: Optional input for latents you want the resulting latent to avoid characteristics of (e.g., "disliked" images).
• pos_weight: The weight of the positive latents.
• neg_weight: The weight of the negative latents.
• null_pos: Conditioning for computing the hidden states of the positive latents, typically an empty clip text embedding.
• null_neg: Similar to null_pos but for negative latents.
• feedback_start: The step to start applying feedback.
• feedback_end: The step to stop applying feedback.
• feedback_percent: The percentage of steps to apply feedback.

ComfyUI fabric Models

ComfyUI_fabric does not introduce new models but rather enhances existing ones by adding the capability to incorporate iterative feedback. This means you can use your favorite diffusion models and enhance them with FABRIC's feedback mechanism.

What's New with ComfyUI fabric

Updates and Changes

• Initial Release: Introduced nodes for integrating FABRIC into ComfyUI.
• Advanced Patch Model: Added support for null_pos and null_neg inputs.
• Batch Latents Node: Facilitates the use of multiple positive/negative latents. These updates make it easier to customize and refine your generative process, providing more control over the final output.

Troubleshooting ComfyUI fabric

Common Issues and Solutions

1. Out of Memory Errors:

• Solution: Try switching cross-attention methods or use a smaller batch of positive/negative latents. You can also try the nightly version of PyTorch for better memory efficiency.

2. Latent Size Mismatch:

• Solution: Ensure that input latent, pos_latents, and neg_latents are all the same size. If they are not, resize them in pixel space or use a model to resize the latents.

3. Feedback Weight Issues:

• Solution: Adjust the pos/neg weights. The default value of 1.0 is often too high; try lowering it.

Frequently Asked Questions

• Q: What should I do if the feedback process is too complex?
• A: The custom nodes available here can simplify the feedback process.
• Q: How should I set the feedback start and end steps?
• A: Researchers recommend applying feedback only during the first half of the denoising steps.

Learn More about ComfyUI fabric

For additional resources, tutorials, and community support, you can explore the following:

• FABRIC Project Page
• Colab Notebook
• Gradio Demo These resources provide a wealth of information to help you get the most out of ComfyUI_fabric and enhance your AI artistry.