MMAudio | Video-to-Audio

The nodes and its associated workflow are fully developed by Kijai. We give all due credit to Kijai for this innovative work. On the RunComfy platform, we are simply presenting Kijai’s contributions to the community. It is important to note that there is currently no formal connection or partnership between RunComfy and Kijai. We deeply appreciate Kijai’s work!

MMAudio

MMAudio is a powerful tool for creating synchronized audio from video and text inputs. It utilizes multimodal joint training to learn from diverse audio-visual and audio-text datasets, ensuring exceptional adaptability. With its advanced synchronization module, it perfectly aligns audio to video frames. MMAudio revolutionizes audio generation, streamlining the process for creators and innovators alike.

1.1 How to Use MMAudio Workflow?

This is the MMAudio workflow, Left Side nodes are inputs for uploading video, Middle is processing MMAudio nodes, and right is the outputs node.

• Upload your Video in input nodes.
• Write your audio generation prompts.
• Click Render !!!

1.2 Video Input

• Click and Upload your Reference Video.

The video is set to downscale the video to ?*512 resolution as processing HD Video or longer video may run of out memory.

1.3 MMAudio Processing

• Positive: Enter the video generation prompts for the audio.
• Negative: Enter what you don't want to hear.
• Steps : More steps may improve audio quality.

1.4 MMAudio Models

These are the model downloader nodes, it will automatically download models in your comfyui in 2-3 mins.

• MMAudio Models : https://github.com/hkchengrex/MMAudio

With its innovative multimodal training and precise synchronization, MMAudio sets a new standard in audio generation. Whether you're crafting videos, animations, or immersive experiences, MMAudio empowers creators with seamless, high-quality audio. Elevate your projects and bring your ideas to life with MMAudio.