ComfyUI Extension: ComfyUI_VisualStylePrompting

ComfyUI_VisualStylePrompting Introduction

ComfyUI_VisualStylePrompting is an extension designed to enhance your AI-generated images by applying specific visual styles. This extension leverages the concept of "Visual Style Prompting with Swapping Self-Attention," developed by Naver AI, to allow you to create images that maintain a consistent style without the need for extensive fine-tuning. Whether you're looking to add a unique artistic touch or ensure stylistic consistency across a series of images, this extension can help you achieve your creative goals.

How ComfyUI_VisualStylePrompting Works

At its core, ComfyUI_VisualStylePrompting works by swapping certain features between your reference image and the generated image during the denoising process. Imagine you have a reference image with a specific style you love, like a painting with deep colors and intricate shading. This extension allows you to transfer those stylistic elements to your new image while keeping the content described by your text prompt.

Here's a simplified breakdown:

1. Reference Latent: The visual style from your reference image is encoded.
2. Positive Conditioning: Describes the desired output image.
3. Swapping Self-Attention: During the image generation process, the extension swaps certain features from the reference image into the new image, ensuring the style is transferred effectively.

ComfyUI_VisualStylePrompting Features

Key Features:

• Reference Latent: Use a VAE-encoded image as a style reference.
• Positive Conditioning: Input text prompts that describe the desired output image.
• Reference Conditioning: Input text prompts that describe the reference image.
• Enabled: Toggle the effect on or off.
• Denoise: Similar to Img2Img, but works with reference and/or initial images.
• Input Blocks: Focuses attention on the encoder layers.
• Skip Input Layers: Specify the number of layers in the input block to exclude from swapping self-attention.
• Middle Block: Focuses attention on the middle layers.
• Skip Middle Layers: Specify the number of layers in the middle block to exclude from swapping self-attention.
• Output Blocks: Focuses attention on the decoder layers.
• Skip Output Layers: Specify the number of layers in the output block to exclude from swapping self-attention.

Customization:

• Positive and Reference Conditioning: Craft detailed prompts to fine-tune the style transfer. For example, use "orange fox, origami, deep colors, shading, canon 60d" for the positive conditioning and "origami figurine" for the reference conditioning to achieve a specific artistic effect.
• Layer Skipping: Adjust the number of layers to skip in each block to control how much of the reference style is applied.

ComfyUI_VisualStylePrompting Models

Currently, this extension has been tested with 1.5-based models. These models are designed to work seamlessly with the Visual Style Prompting technique, ensuring high-quality and consistent results.

What's New with ComfyUI_VisualStylePrompting

Updates:

• Major Changes: Significant updates have been made to improve functionality. Ensure you update your workflows accordingly.
• Experimental Denoise Feature: A new denoise feature has been added, which works similarly to Img2Img but is tailored for use with reference and initial images.

Troubleshooting ComfyUI_VisualStylePrompting

Common Issues and Solutions:

1. Effect Not Visible:

• Ensure the "enabled" toggle is turned on.
• Verify that your reference latent and conditioning prompts are correctly set.

2. Unexpected Outputs:

• Double-check your positive and reference conditioning prompts for accuracy.
• Experiment with different layer skipping settings to fine-tune the style transfer.

3. Multiple Outputs:

• Due to the method's iterative denoising process, you may get two outputs. Use a Batch Index Select node to remove the reference latent from the output.

FAQs:

• Q: Can I use this with models other than 1.5-based models?
• A: Currently, it has only been tested with 1.5-based models. Using other models may yield unpredictable results.
• Q: How do I get the best results?
• A: Focus on crafting detailed and accurate positive and reference conditioning prompts. Adjust the layer skipping settings to control the extent of style transfer.

Learn More about ComfyUI_VisualStylePrompting

For additional resources, tutorials, and community support, consider exploring the following:

• Visual Style Prompting Project Page
• Visual Style Prompting Paper
• HuggingFace Demo
• HuggingFace Demo with ControlNet These resources provide in-depth information and examples to help you make the most of ComfyUI_VisualStylePrompting.