ComfyUI Node: LivePortraitProcess

LivePortraitProcess:

The LivePortraitProcess node is designed to create dynamic and realistic animated portraits from static images. This node leverages advanced AI techniques to animate facial features based on driving images or videos, allowing you to bring still portraits to life. The primary goal of this node is to provide a seamless and intuitive way to generate animated portraits that can mimic expressions and movements from a reference video or image sequence. This is particularly useful for AI artists looking to add a layer of interactivity and realism to their digital art projects. The node handles various complex processes such as image cropping, keypoint extraction, feature transformation, and retargeting of facial expressions, making it a powerful tool for creating lifelike animations with minimal manual intervention.

LivePortraitProcess Input Parameters:

source_image

The source_image parameter is the static portrait image that you want to animate. This image serves as the base for all subsequent animations. The quality and resolution of this image can significantly impact the final output, so it is recommended to use high-resolution images for best results.

driving_images

The driving_images parameter consists of a sequence of images or a video that provides the motion and expressions to be transferred to the source image. These driving images dictate how the facial features of the source image will move and animate. The more expressive and varied the driving images, the more dynamic the resulting animation will be.

dsize

The dsize parameter specifies the desired size of the output image. This parameter allows you to control the resolution of the animated portrait, ensuring it fits your specific requirements. The value should be provided in the format (width, height).

scale

The scale parameter adjusts the scaling factor for the image processing pipeline. This can be used to fine-tune the size of the facial features and overall image to better match the driving images. The default value is typically 1.0, meaning no scaling.

vx_ratio

The vx_ratio parameter controls the horizontal scaling ratio for the image. This can be useful for adjusting the width of the facial features to better align with the driving images. The value is usually between 0.0 and 1.0.

vy_ratio

The vy_ratio parameter controls the vertical scaling ratio for the image. Similar to vx_ratio, this parameter helps in adjusting the height of the facial features. The value is usually between 0.0 and 1.0.

pipeline

The pipeline parameter is an instance of the LivePortraitPipeline class, which orchestrates the entire animation process. This includes tasks like cropping, keypoint extraction, and feature transformation. This parameter is essential for the node to function correctly.

lip_zero

The lip_zero parameter is a boolean flag that, when set to true, initializes the lip-open scalar to zero. This can be useful for ensuring that the mouth remains closed at the start of the animation. The default value is false.

eye_retargeting

The eye_retargeting parameter is a boolean flag that enables or disables eye retargeting. When enabled, the node will adjust the eye movements to better match the driving images. The default value is false.

lip_retargeting

The lip_retargeting parameter is a boolean flag that enables or disables lip retargeting. When enabled, the node will adjust the lip movements to better match the driving images. The default value is false.

stitching

The stitching parameter is a boolean flag that enables or disables the stitching of the animated portrait back onto the original image. This can be useful for blending the animated features seamlessly with the original portrait. The default value is false.

relative

The relative parameter is a boolean flag that, when set to true, makes the animation relative to the initial pose of the source image. This can help in maintaining a more natural look. The default value is false.

eyes_retargeting_multiplier

The eyes_retargeting_multiplier parameter is a scalar value that multiplies the eye retargeting adjustments. This allows for fine-tuning the intensity of the eye movements. The default value is typically 1.0.

lip_retargeting_multiplier

The lip_retargeting_multiplier parameter is a scalar value that multiplies the lip retargeting adjustments. This allows for fine-tuning the intensity of the lip movements. The default value is typically 1.0.

onnx_device

The onnx_device parameter specifies the device to be used for ONNX model inference. Common options include 'CPU' and 'CUDA' for GPU acceleration. The default value is 'CUDA'.

LivePortraitProcess Output Parameters:

cropped_out_list

The cropped_out_list parameter is a list of cropped images that have been processed through the pipeline. These images are intermediate results that show the source image after various transformations like cropping and scaling.

full_out_list

The full_out_list parameter is a list of fully processed images that have been animated and possibly stitched back onto the original portrait. These images represent the final output of the node, showcasing the animated portrait in its entirety.

LivePortraitProcess Usage Tips:

• Ensure that the source image is of high quality and resolution to achieve the best animation results.
• Use expressive and varied driving images to create more dynamic and lifelike animations.
• Fine-tune the dsize, scale, vx_ratio, and vy_ratio parameters to better match the source image with the driving images.
• Enable eye_retargeting and lip_retargeting for more accurate facial animations, especially if the driving images have significant eye and lip movements.
• Utilize the stitching parameter to blend the animated features seamlessly with the original portrait, creating a more cohesive final output.

LivePortraitProcess Common Errors and Solutions:

"Invalid source image format"

• Explanation: The source image provided is not in a supported format.
• Solution: Ensure that the source image is in a common format like JPEG or PNG and try again.

"Driving images sequence is empty"

• Explanation: The driving images parameter is empty or not properly loaded.
• Solution: Verify that the driving images are correctly loaded and passed to the node. Ensure they are in a supported format and try again.

"ONNX model inference failed"

• Explanation: The ONNX model could not be loaded or executed on the specified device.
• Solution: Check that the ONNX model files are correctly placed and accessible. Ensure that the specified device (CPU or CUDA) is available and properly configured.

"Mismatch in image dimensions"

• Explanation: The dimensions of the source image and driving images do not match the expected sizes.
• Solution: Adjust the dsize, vx_ratio, and vy_ratio parameters to ensure that the images are correctly resized and aligned.