Clay Style with Unsampling

This workflow, created by Inner-Reflections, explores Unsampling as a method for consistent style transfer in ComfyUI. For a detailed guide on how to use Unsampling, refer to: Consistent Style Transfer with Unsampling.

By using this workflow, you can transform your video into a clay style animation. The process is simple, requiring minimal effort to achieve a seamless and striking claymation look, reminiscent of traditional stop-motion clay animations.

The Unsampling process in ComfyUI converts the input video into latent noise that preserves the original motion and structure. Unlike random noise, this representational noise serves as the foundation for the diffusion process, enabling the AI to apply the clay style while maintaining temporal consistency across the video frames.

Key Components of the Clay Style Unsampling Workflow:

Inputs: This is where you upload your input video. The frame_load_cap setting limits the number of frames loaded from the input. The select_every_nth setting skips frames to reduce the overall workload. Increasing the select_every_nth value speeds up the process but may affect the smoothness of the final clay style animation.

Unsample: The Unsample component utilizes euler sampling to find noise that accurately represents the input video. By adjusting the cfg scales, you can fine-tune the Unsampling process to match the input more closely. The Flip Sigma node reverses the diffusion process, converting the input frames into latent noise.

Prompt: A well-constructed prompt can significantly improve the final result, especially when aiming for a distinctive clay style. For this style, a good positive prompt might be "(claymation:1.15), stop motion, (everything made of clay:1.1), beautiful woman dancing." This prompt guides the model to recreate the animation in a way that mimics real clay movements and textures.

ControlNet: The ControlNet component uses Depth ControlNet, preprocessed by DepthAnythingV2, to maintain the depth and geometry of the original video. This ensures that the resampling process adheres to the structure of the input video. The strength setting controls how much influence ControlNet has on the output, with higher values leading to more precise structure retention.

Resample: The Resample component takes the unsampled latent noise and runs the diffusion process again, applying the clay style to generate a smooth and consistent claymation effect.

Outputs: The final output frames are decoded, saved as individual images, and then combined into an upscaled MP4 video, preserving the original frame rate while showcasing the new clay style effect.

The clay style is just one of many exciting possibilities you can achieve with this Unsampling workflow in ComfyUI. Experiment with different prompts and settings to create your unique style, and enjoy the process of transforming your videos into dynamic and visually stunning animations!