ComfyUI Node: MarigoldDepthEstimationVideo

MarigoldDepthEstimationVideo:

The MarigoldDepthEstimationVideo node is designed to provide high-quality depth estimation for video sequences using a diffusion-based monocular depth estimation method. This node is particularly beneficial for ensuring consistency between frames in a video, leveraging optical flow to achieve smooth and coherent depth maps across the sequence. By integrating advanced techniques such as ensembling and optical flow, this node aims to deliver accurate and visually appealing depth maps, making it an essential tool for AI artists working on video projects that require depth information. The node supports various configurations to balance between processing time and accuracy, allowing you to tailor the depth estimation process to your specific needs.

MarigoldDepthEstimationVideo Input Parameters:

denoise_steps

This parameter determines the number of steps per depth map during the denoising process. Increasing the number of denoise steps can enhance the accuracy of the depth map but will also increase the processing time. The minimum value is 1, and there is no strict maximum, but higher values will require more computational resources. The default value is typically set to a moderate number to balance accuracy and performance.

n_repeat

This parameter specifies the number of iterations to be ensembled into a single depth map. A higher number of repeats can improve the quality and stability of the depth map by averaging out noise and inconsistencies. The minimum value is 1, and there is no strict maximum, but higher values will increase the processing time. The default value is usually set to a reasonable number to ensure good quality without excessive computation.

n_repeat_batch_size

This parameter defines how many of the n_repeats are processed as a batch. If you have sufficient VRAM, setting this value to match n_repeats can speed up the processing. The minimum value is 1, and the maximum value is typically limited by your hardware capabilities. The default value is set to a lower number to accommodate systems with limited VRAM.

model

This parameter allows you to choose between the Marigold model and its LCM version, marigold-lcm-v1-0. The LCM model is optimized for faster processing with fewer steps, typically around 4 steps, and works best with the LCMScheduler. The default model is Marigold, which provides a good balance between speed and accuracy.

scheduler

Different schedulers can be used to achieve slightly different results in the depth estimation process. This parameter allows you to select the scheduler that best fits your needs. The default scheduler is chosen to provide a good balance of performance and quality.

invert

By default, the Marigold model produces depth maps where black represents the front. This parameter allows you to invert the depth map so that white represents the front, which is often required for compatibility with controlnets and other applications. The default value is false, meaning no inversion.

regularizer_strength

This parameter controls the strength of the regularizer used in the ensembling process. It is generally recommended not to adjust this parameter unless you have specific requirements. The default value is set to ensure stable and high-quality depth maps.

reduction_method

This parameter specifies the method used to reduce the ensembled depth maps into a single output. It is generally recommended not to adjust this parameter unless you have specific requirements. The default value is chosen to provide the best balance of quality and performance.

max_iter

This parameter sets the maximum number of iterations for the ensembling process. It is generally recommended not to adjust this parameter unless you have specific requirements. The default value is set to ensure the process completes in a reasonable time while maintaining quality.

tol

This parameter defines the tolerance level for the ensembling process. It is generally recommended not to adjust this parameter unless you have specific requirements. The default value is set to ensure stable and high-quality depth maps.

use_fp16

This parameter allows you to choose between using fp16 and fp32 precision. Using fp16 can significantly reduce VRAM usage but may lead to a loss of quality in some cases. The default value is false, meaning fp32 is used for better quality.

MarigoldDepthEstimationVideo Output Parameters:

ensembled_image

The ensembled_image output is the final depth map generated by the node. This image represents the depth information of the video frames, with pixel values indicating the distance from the camera. The depth map can be used for various applications, such as 3D reconstruction, augmented reality, and more. The output is typically a high-quality, consistent depth map that has been processed to ensure smooth transitions between frames in a video sequence.

MarigoldDepthEstimationVideo Usage Tips:

• For best results, start with the default settings and gradually adjust the denoise_steps and n_repeat parameters to find the optimal balance between processing time and depth map quality.
• If you have a powerful GPU with ample VRAM, increase the n_repeat_batch_size to speed up the processing.
• Use the invert parameter if you need the depth map to be compatible with controlnets or other applications that require white to represent the front.
• Experiment with different schedulers to see which one provides the best results for your specific video sequence.

MarigoldDepthEstimationVideo Common Errors and Solutions:

"CUDA out of memory"

• Explanation: This error occurs when the GPU runs out of memory while processing the depth maps.
• Solution: Reduce the n_repeat_batch_size or use_fp16 to lower the VRAM usage. Alternatively, try reducing the resolution of the input video frames.

"Invalid model selected"

• Explanation: This error occurs when an unsupported model name is provided in the model parameter.
• Solution: Ensure that the model parameter is set to either "Marigold" or "marigold-lcm-v1-0".

"Depth map values out of range"

• Explanation: This error occurs when the depth map values are not within the expected range.
• Solution: Check the invert parameter and ensure it is set correctly. Also, verify that the input video frames are properly preprocessed.

"Scheduler not found"

• Explanation: This error occurs when an unsupported scheduler name is provided.
• Solution: Ensure that the scheduler parameter is set to a valid scheduler name supported by the node.