ComfyUI Extension: ComfyUI-IC-Light-Native

ComfyUI-IC-Light-Native Introduction

ComfyUI-IC-Light-Native is a native implementation of the IC-Light project within the ComfyUI framework. This extension allows AI artists to manipulate the illumination of images, providing tools to relight images in a consistent and controlled manner. By integrating IC-Light into ComfyUI, the author has made it easier for users to apply advanced lighting effects to their images without needing extensive technical knowledge.

Key Features:

• Relighting Models: Use pre-trained models to relight images based on text or background conditions.
• Foreground and Background Workflows: Separate workflows for handling foreground (FG) and background (BG) elements.
• Detail Preservation: Tools to maintain high-frequency details in images during the relighting process.

How ComfyUI-IC-Light-Native Works

ComfyUI-IC-Light-Native works by leveraging pre-trained models to adjust the lighting of images. The extension uses a process called "latent diffusion" to apply lighting changes in a consistent manner. This means that the lighting effects are applied in a way that maintains the natural appearance of the image, even when different light sources are combined.

Basic Principles:

1. Latent Diffusion: This technique involves modifying the latent space (a compressed representation of the image) to apply lighting changes.
2. Foreground and Background Separation: The extension can handle foreground and background elements separately, allowing for more precise control over the lighting.
3. Detail Transfer: High-frequency details from the original image can be preserved and transferred to the relit image, ensuring that important features are not lost.

ComfyUI-IC-Light-Native Features

Relighting Models

• iclight_sd15_fc_unet_ldm: Use this model for foreground workflows. It is designed to handle images where the main subject is in the foreground.
• iclight_sd15_fbc_unet_ldm: Use this model for background workflows. It is suitable for images where the background needs to be relit.

Customization Options

• Light Maps: Create and apply custom light maps to control the direction and intensity of the lighting.
• Detail Transfer: Use the DetailTransfer node to preserve high-frequency details from the input image.

Example Workflows

• Given FG, Generate BG and Relight: This workflow generates a background based on the given foreground and applies relighting.
• Given FG and Light Map, Generate BG and Relight: This workflow uses a custom light map to control the lighting direction.
• Given FG and BG, Combine and Relight: This workflow combines a given foreground and background and applies relighting.
• Recover High-Frequency Detail: This workflow focuses on preserving and transferring high-frequency details from the original image.

ComfyUI-IC-Light-Native Models

Available Models

• iclight_sd15_fc_unet_ldm: Best used in foreground workflows where the main subject is in the foreground.
• iclight_sd15_fbc_unet_ldm: Ideal for background workflows where the background needs to be relit.

Model Usage

• Foreground Model (iclight_sd15_fc_unet_ldm): Use this model when you want to relight the main subject of the image.
• Background Model (iclight_sd15_fbc_unet_ldm): Use this model when the background of the image needs to be adjusted.

Troubleshooting ComfyUI-IC-Light-Native

Common Issues and Solutions

1. Model Loading Errors:

• Error: RuntimeError: Given groups=1, weight of size [320, 4, 3, 3], expected input[2, 8, 64, 64] to have 4 channels, but got 8 channels instead
• Solution: Ensure that the IC-Light's unet is properly loaded by installing ComfyUI-layerdiffuse.
• Error: RuntimeError: Given groups=1, weight of size [320, 8, 3, 3], expected input[2, 12, 64, 64] to have 8 channels, but got 12 channels instead
• Solution: Verify that you are using the correct model for the workflow (FG or BG).

Frequently Asked Questions

• Q: Why is my background obscured or darkened?
• A: Ensure that the masked/transparent areas of the foreground image are grey before passing it to the VAE. Use the IC Light Apply Mask Grey node to correct the masked area's color.

Learn More about ComfyUI-IC-Light-Native

Additional Resources

• ComfyUI-layerdiffuse: Required for IC-Light to work properly.
• ComfyUI-KJNodes: Provides various mask nodes to create light maps.
• ComfyUI-Easy-Use: A comprehensive node pack that includes tools for background removal and more.
• ComfyUI-IC-Light: Another implementation of IC-Light with useful nodes like DetailTransfer. By exploring these resources, you can gain a deeper understanding of how to use ComfyUI-IC-Light-Native and enhance your AI art projects with advanced lighting effects.