ComfyUI Node: MarigoldDepthEstimation_v2_video

MarigoldDepthEstimation_v2_video:

MarigoldDepthEstimation_v2_video is an advanced node designed for diffusion-based monocular depth estimation in video sequences. This node leverages optical flow to ensure consistency between frames, making it particularly useful for applications requiring smooth and coherent depth maps across video frames. By integrating the Marigold model, it provides high-quality depth estimation, which is essential for various AI art and video processing tasks. The node is experimental and offers several parameters to fine-tune the depth estimation process, balancing between accuracy and processing time. It is ideal for artists and developers looking to enhance their video projects with precise depth information.

MarigoldDepthEstimation_v2_video Input Parameters:

denoise_steps

This parameter defines the number of steps per depth map. Increasing the number of steps can improve the accuracy of the depth estimation but will also increase the processing time. The minimum value is 1, and there is no strict maximum, but higher values will require more computational resources. The default value is typically set to balance accuracy and performance.

n_repeat

This parameter specifies the number of iterations to be ensembled into a single depth map. More iterations can lead to a more refined depth map but will also increase the processing time. The minimum value is 1, and the maximum value depends on the available VRAM. The default value is usually set to provide a good balance between quality and speed.

n_repeat_batch_size

This parameter determines how many of the n_repeats are processed as a batch. If you have sufficient VRAM, setting this value to match n_repeats can speed up the processing. The minimum value is 1, and the maximum value is typically the same as n_repeats. The default value is set based on the available VRAM.

model

This parameter allows you to choose between the Marigold model and its LCM version, marigold-lcm-v1-0. The LCM model is optimized for fewer steps and works best with the LCMScheduler. The default model is Marigold.

scheduler

Different schedulers can produce slightly different results. This parameter lets you select the scheduler that best fits your needs. The default scheduler is chosen to provide a good balance between speed and quality.

invert

By default, Marigold produces a depth map where black represents the front. This parameter allows you to invert the depth map, which is useful for applications like controlnets that require the opposite representation. The default value is false.

regularizer_strength

This parameter controls the strength of the regularizer in the ensembling process. It is generally recommended not to change this value unless you have specific requirements. The default value is set to provide stable results.

reduction_method

This parameter specifies the method used for reducing the depth maps during the ensembling process. It is generally recommended not to change this value. The default method is chosen to ensure high-quality depth maps.

max_iter

This parameter sets the maximum number of iterations for the ensembling process. It is generally recommended not to change this value. The default value is set to ensure the process completes in a reasonable time.

tol

This parameter defines the tolerance for the ensembling process. It is generally recommended not to change this value. The default value is set to provide stable and accurate results.

use_fp16

This parameter determines whether to use fp16 (half-precision) or fp32 (single-precision) for computations. Using fp16 can significantly reduce VRAM usage but may lead to a loss of quality in some cases. The default value is false, meaning fp32 is used.

MarigoldDepthEstimation_v2_video Output Parameters:

ensembled_image

This output parameter provides the final ensembled depth map image. The depth map is a high-quality representation of the depth information in the video frames, which can be used for various applications such as 3D reconstruction, augmented reality, and AI art projects. The depth values are normalized between 0 and 1, where 0 represents the farthest point and 1 represents the nearest point.

MarigoldDepthEstimation_v2_video Usage Tips:

• For higher accuracy, increase the denoise_steps parameter, but be aware that this will also increase the processing time.
• If you have sufficient VRAM, set the n_repeat_batch_size to match n_repeat for faster processing.
• Use the invert parameter if your application requires a depth map where white represents the front.
• Experiment with different schedulers to find the one that best fits your specific needs and produces the desired results.

MarigoldDepthEstimation_v2_video Common Errors and Solutions:

"CUDA out of memory"

• Explanation: This error occurs when the GPU runs out of memory while processing the depth estimation.
• Solution: Reduce the n_repeat_batch_size or denoise_steps parameters to lower the memory usage. Alternatively, use the use_fp16 parameter to reduce VRAM consumption.

"Invalid depth map values"

• Explanation: This error occurs when the depth map contains invalid values outside the expected range.
• Solution: Ensure that the input images are correctly preprocessed and that the parameters are set within reasonable ranges. Check for any anomalies in the input data.

"Model not found"

• Explanation: This error occurs when the specified model is not available or incorrectly specified.
• Solution: Verify that the model name is correct and that the model files are properly installed and accessible. Use the default model if unsure.

"Scheduler not supported"

• Explanation: This error occurs when an unsupported scheduler is selected.
• Solution: Choose a scheduler that is supported by the MarigoldDepthEstimation_v2_video node. Refer to the documentation for a list of supported schedulers.