ComfyUI Node: Storydiffusion_Img2Img

Storydiffusion_Img2Img:

Storydiffusion_Img2Img is a powerful node designed to transform an existing image by applying various artistic styles and character prompts to create a new, visually compelling output. This node leverages advanced diffusion models to blend the original image with specified character and scene prompts, allowing you to generate unique and creative images that align with your artistic vision. Whether you are looking to enhance an image with specific stylistic elements or create a narrative-driven visual, Storydiffusion_Img2Img provides the tools to achieve these goals. The node is particularly useful for AI artists who want to experiment with different styles and prompts to produce diverse and imaginative artwork.

Storydiffusion_Img2Img Input Parameters:

image

This parameter accepts the original image that you want to transform. The image serves as the base for applying the specified styles and prompts.

pipe

This parameter requires a pre-trained model pipeline that will be used for the image transformation process. The model should be compatible with the diffusion techniques employed by the node.

info

A string parameter that contains metadata about the model and its configuration. This includes details like model type, checkpoint path, LoRA path, original config file, LoRA scale, and other relevant information. This metadata is crucial for ensuring the correct application of the model settings.

character_prompt

A multiline string parameter where you can specify character descriptions that will influence the transformation. For example, [Taylor]a woman img, wearing a white T-shirt, blue loose hair.\n[Lecun] a man img,wearing a suit,black hair. These prompts help in defining the characters that will be integrated into the image.

scene_prompts

A multiline string parameter for specifying scene descriptions that will guide the overall style and context of the transformed image. This allows for a more narrative-driven approach to image generation.

negative_prompt

A string parameter for specifying elements that should be avoided in the transformation. This helps in refining the output by excluding unwanted features.

img_style

A string parameter that defines the artistic style to be applied to the image. This can include various predefined styles or custom styles as per your requirement.

seed

An integer parameter that sets the random seed for the transformation process. This ensures reproducibility of the results. Default value is typically set to a random seed.

steps

An integer parameter that defines the number of diffusion steps to be applied. More steps generally result in higher quality images but take longer to process. The default value is 50, with a minimum of 1 and a maximum of 1024.

cfg

A float parameter that controls the classifier-free guidance scale. This influences the strength of the guidance applied during the transformation. Default value is 7.5, with a minimum of 1.0 and a maximum of 20.0.

ip_adapter_strength

A float parameter that adjusts the strength of the image projection adapter. This affects how strongly the original image features are preserved. Default value is 0.5, with a minimum of 0.0 and a maximum of 1.0.

style_strength_ratio

An integer parameter that determines the ratio of style strength applied to the image. Default value is 20, with a minimum of 10 and a maximum of 50.

encoder_repo

A string parameter specifying the repository of the encoder model to be used. Default value is laion/CLIP-ViT-bigG-14-laion2B-39B-b160k.

role_scale

A float parameter that adjusts the scale of the role played by the character prompts in the transformation. Default value is 0.8, with a minimum of 0.0 and a maximum of 1.0.

mask_threshold

A float parameter that sets the threshold for masking during the transformation. This helps in refining the areas of the image that are affected by the prompts. Default value is 0.5, with a minimum of 0.0 and a maximum of 1.0.

start_step

An integer parameter that defines the starting step for the diffusion process. This can be used to skip initial steps and start the transformation from a later stage. Default value is 5, with a minimum of 1 and a maximum of 1024.

Storydiffusion_Img2Img Output Parameters:

image

The transformed image that incorporates the specified styles and prompts. This output is the final visual result of the diffusion process.

prompt_array

A string array that contains the prompts used during the transformation. This helps in understanding the elements that influenced the final image.

Storydiffusion_Img2Img Usage Tips:

• Experiment with different character and scene prompts to see how they influence the final image. This can help you discover unique and creative combinations.
• Adjust the style_strength_ratio to balance between preserving the original image features and applying the new style.
• Use the negative_prompt parameter to exclude unwanted elements and refine the output.
• Set a specific seed value if you want to reproduce the same results in future transformations.

Storydiffusion_Img2Img Common Errors and Solutions:

"Invalid model pipeline"

• Explanation: The provided model pipeline is not compatible with the node.
• Solution: Ensure that you are using a pre-trained model pipeline that supports the diffusion techniques used by the node.

"Metadata parsing error"

• Explanation: The info string parameter contains incorrect or improperly formatted metadata.
• Solution: Verify that the info string is correctly formatted and includes all required details like model type, checkpoint path, LoRA path, etc.

"Image size mismatch"

• Explanation: The input image dimensions do not match the expected size for the transformation process.
• Solution: Ensure that the input image meets the required dimensions specified by the model pipeline.

"Invalid seed value"

• Explanation: The seed value provided is not an integer.
• Solution: Make sure to provide an integer value for the seed parameter to ensure reproducibility.

"Step value out of range"

• Explanation: The number of steps specified is outside the allowed range.
• Solution: Adjust the steps parameter to be within the range of 1 to 1024.